Advanced on-orbit system intelligent framing method based on near-end strategy optimization

Abstract

The invention discloses an advanced on-orbit system intelligent framing method based on near-end strategy optimization, which comprises the following steps: establishing a data packet model, enabling a data packet of a real-time service to enter a time-sensitive service queue, and enabling a data packet of a non-real-time service to enter a non-time-sensitive service queue; and executing a real-time framing algorithm and an efficient framing algorithm for data packets in the time-sensitive service queue and the non-time-sensitive service queue, calculating average framing time, frame multiplexing efficiency and average packet delay, packaging the data packets into transmission frames, and enabling the packaged transmission frames to enter a virtual channel scheduling unit and to be output. According to the method designed by the invention, different transmission requirements of real-time services and non-real-time services are considered, the network service quality of the AOS system is enhanced, and efficient transmission of spatial data with diversified service types is realized.

Description

Technical Field

[0001] The present invention relates to the field of information engineering, and in particular to an advanced on-orbit system intelligent framing method based on proximal strategy optimization. Background Art

[0002] In recent years, space exploration activities such as deep space exploration, manned space flight, and satellite development have become increasingly active around the world. The returned data involves engineering telemetry, physiological telemetry, engineering remote control, scientific observation, scientific experiments, time-delay playback, audio, video, etc., showing new characteristics such as large differences in business types, large differences in transmission rates, and large differences in service quality (QoS) requirements, which puts higher requirements on the real-time and reliability of space data system transmission. To this end, the Consultative Committee for Space Data Systems (CCSDS) has formulated the Advanced Orbiting System (AOS) protocol. AOS requires that multiple data packets with the same or similar QoS can be reused in the same virtual channel scheduling unit, and segmented into a fixed-length Multiplexing Protocol Data Unit (MPDU), and a complete data frame is generated after adding control information to improve the efficiency of space link multiplexing.

[0003] Currently, the frame generation algorithms in AOS are mainly divided into three types: isochronous framing algorithm, efficient framing algorithm and adaptive framing algorithm. The principle of the isochronous framing algorithm is to set the framing time threshold to a fixed value. No matter how many data packets arrive, they will be encapsulated into a frame when the set framing time is reached. If the total number of arriving data packets does not reach the MPDU packet area length, idle packets are used for filling. However, for non-time-sensitive data packets, because they do not have high requirements for framing time, the use of isochronous framing algorithms may result in too low frame multiplexing efficiency. Therefore, an efficient framing algorithm is proposed. The principle is that when data packets continuously enter the queue, they are encapsulated into frames and sent only when the cumulative data packet length reaches the length of an MPDU packet area. However, the above two framing algorithm standards are too single, while the adaptive framing algorithm can be controlled by pre-setting the framing time threshold and MPDU packet length. Specifically, when the frame waiting time is less than the set threshold and the MPDU area is filled with arriving data packets, the frame is encapsulated and sent immediately; when the frame waiting time reaches the maximum threshold and the MPDU buffer is not filled, the idle packet is used to complete the entire transmission frame. However, most of these algorithms require pre-setting of relevant parameters, such as the framing time threshold, the number of iterations, etc., and cannot interact with the environment in real time, resulting in low efficiency in the application of spatial systems with diverse business types and time-varying network states. Summary of the invention

[0004] The purpose of the present invention is to provide an advanced on-orbit system intelligent framing method (AIFG) based on proximal strategy optimization. The method is based on the advanced on-orbit system (AOS) virtual channel multiplexing technology, and aims at the current situation that the existing AOS data services are diverse and complex, the data volume is large, and the existing framing algorithm needs to configure parameters in advance, which is difficult to adapt to the constantly changing network status in a complex network environment. The problem of poor information transmission effect of the framing algorithm is solved, and efficient transmission of space data with diversified service types is achieved.

[0005] To achieve the above functions, the present invention designs an advanced on-orbit system intelligent framing method based on proximal strategy optimization. For the target network, the following steps S1 to S3 are executed to complete the data packet transmission of real-time services and non-real-time services in the network: Step S1: for the arriving data packets, a data packet model about the number of arriving data packets is established based on Poisson distribution; Step S2: define two service types of data packets, including real-time services and non-real-time services. After all data packets arrive, data packets of real-time services enter the time-sensitive service queue, and data packets of non-real-time services enter the non-time-sensitive service queue; Step S3: Execute a real-time framing algorithm for the data packets in the time-sensitive service queue, and execute an efficient framing algorithm for the data packets in the non-time-sensitive service queue, calculate the average framing time, frame multiplexing efficiency, and average packet delay of the time-sensitive service queue and the non-time-sensitive service queue respectively, encapsulate the data packets in the time-sensitive service queue and the non-time-sensitive service queue into transmission frames, and send the encapsulated transmission frames to the virtual channel scheduling unit and output them.

[0006] Beneficial effects: Compared with the prior art, the advantages of the present invention include: In order to transmit time-sensitive services as early as possible, the present invention models the optimal time threshold calculation problem for framing such services as a Markov decision process, and uses the proximal policy optimization (PPO) algorithm to obtain the optimal solution based on performance indicators such as frame multiplexing efficiency, average framing time, and average packet delay. In order to improve the frame multiplexing efficiency of non-delay-sensitive services, the present invention requires such services to fill a frame before transmission. This method takes into account the different transmission requirements of real-time services and non-real-time services, and enhances the network service quality of the AOS system. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] Figure 1 It is a system model diagram of an advanced on-orbit system intelligent framing method based on proximal strategy optimization provided according to an embodiment of the present invention; Figure 2 is a schematic diagram of the overall structure of a model of an optimal framing time threshold algorithm provided according to an embodiment of the present invention; Figure 3 is a topological structure diagram of a simulation system provided according to an embodiment of the present invention; Figure 4 is a comparison diagram of the change of the reward function under four sets of parameter value settings when the framing time threshold value is 5ms according to an embodiment of the present invention; Figure 5 is a comparison diagram of the simulation performance of the AIFG algorithm under the condition that the framing time threshold value is 5ms and four groups of parameter values ​​are set according to an embodiment of the present invention; Figure 6 is a diagram showing the throughput simulation results of a time-sensitive service queue provided in an embodiment of the present invention; Figure 7 is a diagram showing the simulation result of frame multiplexing efficiency of a time-sensitive service queue provided in an embodiment of the present invention; Figure 8 This is a diagram of the average queuing delay simulation results of the time-sensitive service queue provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0008] The present invention will be further described below in conjunction with the accompanying drawings. The following embodiments are only used to more clearly illustrate the technical solution of the present invention, and cannot be used to limit the protection scope of the present invention.

[0009] An advanced on-orbit system intelligent framing method based on proximal strategy optimization provided by an embodiment of the present invention performs the following steps S1 to S3 for a target network to complete data packet transmission of real-time services and non-real-time services in the network: Step S1: for the arriving data packets, a data packet model about the number of arriving data packets is established based on Poisson distribution; Assume that the packet arrival process is subject to the parameters l Poisson distribution, that is, the number of arriving packets is k The probability is as follows: ; In the formula, Indicates the number of arriving packets is k The probability of X Indicates time slot t The number of packets arriving within l represents the Poisson distribution parameter, which represents the time slot in the network t The frequency of data packets arriving within a network, that is, the average packet arrival rate.

[0010] If the length of each data packet is a fixed value l s , the number of packets that can be received by one MPDU is n , then the length of the MPDU packet area is At this point, the arriving data packet can fit neatly into the MPDU data area, avoiding the situation where the data area cannot be filled with a complete data packet.

[0011] Step S2: define two service types of data packets, including real-time services and non-real-time services. After all data packets arrive, data packets of real-time services enter the time-sensitive service queue, and data packets of non-real-time services enter the non-time-sensitive service queue; The real-time services described in step S2 include audio data and video data, and the non-real-time services include telemetry data, remote control data, observation data, and experimental data.

[0012] Step S3: Execute a real-time framing algorithm for the data packets in the time-sensitive service queue, and execute an efficient framing algorithm for the data packets in the non-time-sensitive service queue, calculate the average framing time, frame multiplexing efficiency, and average packet delay of the time-sensitive service queue and the non-time-sensitive service queue respectively, encapsulate the data packets in the time-sensitive service queue and the non-time-sensitive service queue into transmission frames, and send the encapsulated transmission frames to the virtual channel scheduling unit and output them.

[0013] The system model diagram of the advanced on-orbit system intelligent framing method based on proximal strategy optimization designed by the present invention is shown in FIG. Figure 1 ,The system model classifies the services in the AOS system into real-time ,services and non-real-time services, and adopts different ,framing decisions for different types of queues. Real-time framing algorithms are used to ,process time-sensitive service queues, and efficient framing algorithms are used to ,process non-time-sensitive

[0014] The network includes an agent, a reward, a buffer, and an environment. The agent has a strategy network and an evaluation network. The network's own parameters include the average packet arrival rate. l , Iterations G , the proportionality coefficient of the decay rate r , Strategy Network Update Cycle T c , Discount Factor , Clipping parameters c , self-update strategy times n update , policy network learning rate t , the number of time slots sampled in one training iteration N .

[0015] The specific steps of the real-time framing algorithm described in step S3 are as follows: Step S3.1.1: Input the network parameters and create the simulation environment and network topology through the NS-3 platform; Step S3.1.2: Generate data streams that conform to Poisson distribution for real-time services and initialize network parameters i ; Step S3.1.3: Adopt the optimal framing time threshold algorithm based on the PPO (Proximal Policy Optimization) algorithm to obtain the optimal framing time threshold. According to the optimal framing time threshold, set the framing time threshold and the maximum framing length of the data packet of the real-time service. The overall structure diagram of the model of the optimal framing time threshold algorithm is shown in the figure below. Figure 2 The specific steps of the optimal framing time threshold algorithm described in step S3.1.3 are as follows: Step S3.1.3.1: Initialize network parameters i and buffer zones; Step S3.1.3.2: Perform training iterations, dividing the training cycle into several consecutive time slots; t , build environment status s t , the set of environmental states for all time slots is represented as , time slot t Environmental status s t The definition is as follows: ; in, R s represents the average arrival rate of the queue in the time slot, L s Indicates the average queue length in the time slot, which is used to judge the load status of the current round. If the queue length is large, it means that the current queue is in a congested state, the next round should be framed as soon as possible, and the framing time threshold should be reduced. T f Indicates the average framing time of data in a time slot, L f Indicates the average effective frame length of the data frame in the time slot; There is a time-sensitive service queue in the environment, which generates transmission frames in real time. Each transmission frame consists of two parts: the framing time threshold T h , that is, valid data packets and idle data packets arriving within the transmission time window. The environment generates a vector describing the environment state based on the current frame and queue state s t , and transmits the real-time monitored environmental status to the agent in real time; Step S3.1.3.3: Agent receives environment state s t Input, according to the current environment status s t Based on the transition probability Generate Action at , Indicates the current strategy and action a t Corresponding to the framing time threshold, save the action a t and the corresponding action probabilities , represents the old strategy, that is, the strategy function used by the agent at a certain stage in the past; Step S3.1.3.4: After a time slot t , the agent makes decisions based on the state of the environment and the reward function, and the environment performs actions a t , the action set of the current time slot is , with action a t The corresponding framing time threshold is used to perform framing operations. The framing time threshold is used to determine when the time-sensitive data is framed in the queue. The action set by the framing time threshold can act on the environment and interact with it. The range of the action is , T m It is the maximum value of the average queuing delay of time-sensitive services.

[0016] During each training, the agent is trained according to the reward function Get the current reward value r t , indicating the current time slot t Environmental status s t Make a decision and take action a t , given a t The reward function value obtained by feedback after the corresponding frame time threshold is used to evaluate the pros and cons of the current action and can provide an effective signal for the next round of intelligent agent training. The reward function is as follows: ; In the formula, Indicates the environmental status s t Take action a t The reward function is L f is the average effective frame length of the data frame in the time slot (excluding idle packets), T f Indicates the average framing time of data in a time slot, T h is the framing time threshold, α and βare the adjustable weights of the average effective frame length and the average framing time, respectively, and , and d Indicates the weight of the framing delay, representing the minimum and maximum allowed range; After the action is executed, the environment state of the next time slot is entered s t+1 , and pass it to the agent; the agent repeats the above process. By continuously interacting with the environment and optimizing the network, the agent gradually optimizes the decision of the framing time threshold, improves the framing capability of the time-sensitive service queue, and finally meets the system's service quality requirements.

[0017] Step S3.1.3.5: Constructing time slots t Environmental samples , save the environment sample to the buffer; Step S3.1.3.6: When the preset update time slot threshold is reached, the agent extracts environmental samples from the buffer, calculates the loss function based on the policy network and the evaluation network, and uses the difference between the predicted reward and the actual reward as the basis to guide the network parameter optimization and update the network parameters, thereby adjusting the agent's strategy.

[0018] If the current action is not the first action, do the following: For each time slot, when the preset update time slot threshold is reached, the objective function is calculated: ; in, represents the objective function, Indicates time slot t The expected value of Is the current strategy With the old strategy The ratio of is the joint advantage function using the generalized advantage estimation (GAE), It is a hyperparameter in the strategy update process, which is used to limit the magnitude of the strategy update; Update the loss function according to the following formula: ; In the formula, loss represents the loss function, r i Indicates the current action a i The reward value, The current state of the environment calculated by the policy network s i of Q Value function, To evaluate the next environment state calculated by the networkQ Value function, is the discount factor, a The next state s i+1 All possible actions, Is used for target Q Parameters of the target network for value calculation; The mean square error is calculated based on the following formula and the evaluation network is updated: ; In the formula, Is the loss function of the evaluation network, used to update the network parameters i , G t From the time slot t The cumulative reward at the beginning, represents the state of the environment estimated by the value function s t The expected return, N is the number of time slots sampled in one training iteration; From time slot t Starting cumulative rewards G t As follows: ; In the formula, R t Indicates time slot t Instant rewards; In the continuous action space, the standard deviation of the action s It will gradually decay to a minimum value s min , this parameter controls the randomness of action selection. When the action space is sufficient and the time slot is equal to the action standard deviation decay frequency, the action standard deviation is updated based on the following formula: ; in, s t It is t The action standard deviation after the time slot update is: s t+1 It is t +1 action standard deviation after time slot update, is the decay rate, r is the proportionality coefficient of the decay rate, s min is the minimum action standard deviation; the constraint condition for the action standard deviation is: ; The above formula ensures that the action standard deviation will not be lower than the set minimum value, thus ensuring continuous exploration.

[0019] Next, the cumulative reward is calculated and saved to the buffer; Step S3.1.3.7: If the amount of data stored in the buffer exceeds the preset capacity, the buffer is cleared after the policy network parameters are updated to ensure the independence of subsequent training; Step S3.1.3.8: After each round of training, update the environment status and prepare to enter the next round of training, and output the current framing time threshold value as the optimal framing time threshold value.

[0020] Step S3.1.4: If the data packet of the real-time service reaches any one of the framing time threshold or the maximum framing length, the data packet is encapsulated in the MPDU packet area, and the data packets are combined into a data frame for transmission; If the framing time threshold is reached and the maximum framing length is not reached, the idle area is filled with idle packets; Step S3.1.5: Calculate the average framing time, frame multiplexing efficiency and average packet delay of the time-sensitive service queue and output them.

[0021] The specific steps of the efficient framing algorithm described in step S3 are as follows: Step S3.2.1: Input the network parameters and create the simulation environment and network topology through the NS-3 platform; Step S3.2.2: For non-real-time services, generate data streams that conform to the Poisson distribution and initialize network parameters i ; Step S3.2.3: Setting the maximum framing length of the data packet of the non-real-time service; Step S3.2.4: When the data packet reaches the maximum framing length, the data packet is encapsulated in the MPDU packet area, and the data packets are combined into a data frame for transmission; Step S3.2.5: Calculate the average framing time, frame multiplexing efficiency and average packet delay of the non-time-sensitive service queue and output them.

[0022] For time-sensitive service queues, set the maximum number of encapsulated data packets in the MPDU packet area to M , the framing time of the time-sensitive service queue, that is, the time required to generate a frame is t sw , then in the interval Inside, t sw The probability density function of for: ; Frame time t sw Framing time threshold The probability is: ; Average framing time of time-sensitive service queues As follows: ; Frame multiplexing efficiency of time-sensitive service queues As follows: ; In the formula, Indicates that within the time limit T The framing time of the time-sensitive service queue t sw Less than T The probability of l s is the packet length, l ms is the length of the MPDU packet area; Average packet delay of time-sensitive service queue As follows: ; In the formula, T d is the packet delay of the time-sensitive service queue, for Arrive within time n The probability of a packet, for Arrive within time M + x The probability of a packet.

[0023] For non-time-sensitive service queues, perform framing operations on non-real-time service data packets based on the maximum number of data packets in the MPDU packet area; set S w is the framing time of the non-time-sensitive service queue, l is the average packet arrival rate, M The maximum number of packets encapsulated in the MPDU packet area, the average framing time of the non-time-sensitive service queue As follows: ; set up T τ is the packet delay of the non-time-sensitive service queue, i is the number of arriving data packets, then the average packet delay of the non-time-sensitive service queue for: ; Because non-time-sensitive services use an efficient framing algorithm to generate frames, the efficient framing algorithm will only frame and transmit when the data packet fills the MPDU packet area. There will be no idle packets to fill in. The data packets in the transmission frame are all useful information. Therefore, the frame multiplexing efficiency of the non-time-sensitive service queue is 100%.

[0024] The following is an application example of the present invention: A joint simulation platform was built using NS-3 and PyTorch to verify the performance of an advanced on-orbit system intelligent framing method (AIFG) based on proximal strategy optimization designed in the present invention.

[0025] like Figure 3 The figure shows the topology of the simulation system. The simulation system includes 50 sending nodes of space probe payloads and 50 receiving nodes simulating ground terminal equipment; Figure 3 D1, D2, ..., D n represents the 1st to nth sending nodes, T1, T2, ..., T m Represents the 1st to mth receiving nodes. After the payload collects various types of data and performs corresponding data packet encapsulation and processing, the data packets are divided into real-time data packets and non-real-time data packets and sent to the communication satellite respectively, entering the time-sensitive and non-time-sensitive service queues. Then, corresponding to the queues of different service types, different framing algorithms are used to complete the data transmission between communication satellites in a targeted manner. Finally, according to the service type, the satellite sends the data to various terminal devices on the ground for further analysis and processing. In order to simulate the complex space links and various services in the AOS system, the communication satellite node sends the typical collection data and communication information of the aerospace payload to the receiving terminal node. The simulation focuses on the framing algorithm in the system. Based on the random data generated by the ON / OFF source model as the data source, the link is simulated as the link type of the communication satellite. At the same time, the advanced on-orbit system intelligent framing algorithm (AIFG) for differentiated QoS requirements is deployed on the sending communication satellite to transmit data between the two satellite nodes. The network parameters are shown in Table 1. In Table 1, RTFG represents the real-time framing algorithm, IFG represents the isochronous framing algorithm, and HEFG represents the efficient framing algorithm:

[0026] The main simulation parameter settings of the PPO algorithm are shown in Table 2:

[0027] As shown in Table 3, this simulation takes the setting of the framing time threshold value as 5ms as an example, sets the parameters of multiple groups of reward functions and observes the reward function, throughput, average packet delay and frame multiplexing efficiency for each case, and compares and obtains the optimal parameter group value:

[0028] When the framing time threshold is 5ms, the reward value convergence curve changes as shown in the figure. Figure 4 Different parameter settings will lead to different changes in the reward function value. Figure 4 shows the mainstream trend of the reward function, which gradually rises in the early stage of training, and begins to stabilize when the number of iterations is about 40, entering a plateau period, indicating the convergence of the reward function. Among the four selected parameter groups, the growth rate of the convergence function is the fastest in parameter group 3, and the growth rate is slower in parameter group 2. At the same time, the convergence effect of parameter groups 1, 3, and 4 is better, but the convergence effect of parameter group 2 is poor.

[0029] As shown in Figure 5, the simulation performance comparison of the AIFG algorithm under four groups of parameter value settings is shown. As shown in Figure 5, the average packet delay and throughput under parameter group 3 are the best, while the frame multiplexing efficiency of parameter group 4 is the best. Overall, selecting parameter group 3 for AIFG algorithm training is a better choice.

[0030] Table 4 shows the specific values ​​of the reward function parameters under the four groups of framing time thresholds. This simulation sets the optimal reward function parameter group for the four framing time thresholds:

[0031] FIG6 is a diagram showing the throughput simulation result of a time-sensitive service queue. In the figure, AFG represents an adaptive framing algorithm; as shown in FIG6 , the throughput of the AIFG algorithm is generally higher than that of other framing algorithms. When the framing time threshold is 5ms, the throughput of the AIFG algorithm is 466Kbps, which is 78.05% higher than that of the AFG+HEFG and IFG+HEFG algorithms. When the framing time threshold is 10ms, the throughput of the AIFG algorithm is 464Kbps, which is 25.37% higher than that of the AFG+HEFG and IFG+HEFG algorithms. When the framing time threshold is 15ms, the throughput of the AIFG algorithm reaches its maximum value of 606.456Kbps, which is 28.66% higher than that of the AFG+HEFG and IFG+HEFG algorithms. When the framing time threshold is 20ms, the throughput of AIFG is also improved to a certain extent, which is 17.18% higher than that of the AFG+HEFG and IFG+HEFG algorithms. When the latency requirement is low, the AIFG algorithm has a higher throughput improvement rate and better performance change; when the latency requirement is high, the AIFG algorithm can obtain a larger throughput.

[0032] Figure 7 shows the simulation result of the frame multiplexing efficiency of the time-sensitive service queue. Due to the FIFO transmission mechanism, the frame multiplexing efficiency of the queue must be 100%. Compared with the IFG and AFG algorithms, the AIFG algorithm has a higher frame multiplexing efficiency when the delay requirement is small. Under the 5ms framing time threshold, the frame multiplexing efficiency of the AIFG algorithm is 69.25%, which is 92.36% higher than that of the AFG+HEFG and IFG+HEFG algorithms. Under the 10ms framing time threshold, the multiplexing efficiency of the AIFG algorithm is 69.1%, which is 52.47% higher than that of the AFG+HEFG and IFG+HEFG algorithms. Under the 15ms framing time threshold, the frame multiplexing efficiency of the AIFG algorithm is 12.26% higher than that of the AFG+HEFG and IFG+HEFG algorithms. Under the 20ms framing time threshold, the frame multiplexing efficiency of the AIFG algorithm reaches the maximum value - 94.7%. However, compared with the AFG+HEFG and IFG+HEFG algorithms, the frame multiplexing efficiency of the AIFG algorithm is slightly reduced. When the delay requirement is high, the frame multiplexing efficiency growth rate of the AIFG algorithm will decrease, which will be affected by the constraints of the framing time threshold and the fixed value of the AOS transmission frame, but the overall frame multiplexing efficiency value shows an upward trend.

[0033] Figure 8 shows the simulation results of the average queuing delay of the time-sensitive service queue. Figure 8 shows that the AIFG method always shows the lowest delay at all time thresholds, outperforming all other methods. This is especially evident at higher time thresholds (15ms and 20ms), where the delay is significantly smaller. At the same time, the FIFO-based method has a slower processing time due to its inherent characteristics. At the framing time threshold of 5ms, the average queuing delay of the AIFG, AFG+HEFG and IFG+HEFG algorithms is not much different, about 6-7ms. However, under the framing time threshold of 10ms, the average queuing delay of the AIFG method is reduced by 19.39% compared with the AFG+HEFG algorithm and 40.33% compared with the IFG+HEFG algorithm; under the framing time threshold of 15ms, the average queuing delay of the AIFG algorithm is reduced by 37.30% compared with the AFG+HEFG algorithm and 39.62% compared with the IFG+HEFG algorithm; under the framing time threshold of 20ms, the average queuing delay of the AIFG algorithm is reduced by 43.90% compared with the AFG+HEFG algorithm and 43.45% compared with the IFG+HEFG algorithm. When the framing time threshold of AIFG is low (such as 5ms), the queue length usually does not exceed the AFG threshold, so the algorithm performance is similar and the advantage of the AIFG algorithm is not reflected. When the framing time threshold is high, the AIFG algorithm can dynamically optimize the gap between frames, reduce delays and improve overall performance by adapting to changing conditions in the transmission environment. The simulation results verify the adaptability of the AIFG algorithm under different delay requirements and its advantages in reducing the average queuing delay.

[0034] The embodiments of the present invention are described in detail above with reference to the accompanying drawings, but the present invention is not limited to the above embodiments, and various changes can be made within the knowledge scope of ordinary technicians in this field without departing from the purpose of the present invention.

Claims

1. An advanced on-orbit system intelligent framing method based on proximal strategy optimization, characterized in that: For the target network, the following steps S1 to S3 are executed to complete the data packet transmission of the real-time service and the non-real-time service in the network: Step S1: for the arriving data packets, a data packet model about the number of arriving data packets is established based on Poisson distribution; Step S2: define two service types of data packets, including real-time services and non-real-time services. After all data packets arrive, data packets of real-time services enter the time-sensitive service queue, and data packets of non-real-time services enter the non-time-sensitive service queue; Step S3: Execute a real-time framing algorithm for the data packets in the time-sensitive service queue, and execute an efficient framing algorithm for the data packets in the non-time-sensitive service queue, calculate the average framing time, frame multiplexing efficiency, and average packet delay of the time-sensitive service queue and the non-time-sensitive service queue respectively, encapsulate the data packets in the time-sensitive service queue and the non-time-sensitive service queue into transmission frames, and send the encapsulated transmission frames to the virtual channel scheduling unit and output them.

2. According to claim 1, an advanced on-orbit system intelligent framing method based on proximal strategy optimization is characterized in that: The data packet model in step S1 is specifically as follows: ； In the formula, Indicates the number of arriving packets is k The probability of X Indicates time slot t The number of packets arriving within λ Represents the average packet arrival rate.

3. The advanced on-orbit system intelligent framing method based on proximal strategy optimization according to claim 1 is characterized in that: The real-time services described in step S2 include audio data and video data, and the non-real-time services include telemetry data, remote control data, observation data, and experimental data.

4. The advanced on-orbit system intelligent framing method based on proximal strategy optimization according to claim 1 is characterized in that: The specific steps of the real-time framing algorithm described in step S3 are as follows: Step S3.1.1: Input the network parameters and create the network topology; Step S3.1.2: Generate data streams that conform to Poisson distribution for real-time services and initialize network parameters θ ; Step S3.1.3: adopting the optimal framing time threshold algorithm, based on the PPO algorithm, to obtain the optimal framing time threshold, and setting the framing time threshold of the data packet of the real-time service and the maximum framing length according to the optimal framing time threshold; Step S3.1.4: If the data packet of the real-time service reaches any one of the framing time threshold or the maximum framing length, the data packet is encapsulated in the MPDU packet area, and the data packets are combined into a data frame for transmission; If the framing time threshold is reached and the maximum framing length is not reached, the idle area is filled with idle packets; Step S3.1.5: Calculate the average framing time, frame multiplexing efficiency and average packet delay of the time-sensitive service queue and output them.

5. The advanced on-orbit system intelligent framing method based on proximal strategy optimization according to claim 4, characterized in that: The specific steps of the efficient framing algorithm described in step S3 are as follows: Step S3.2.1: Input the network parameters and create the network topology; Step S3.2.2: For non-real-time services, generate data streams that conform to the Poisson distribution and initialize network parameters θ ; Step S3.2.3: Setting the maximum framing length of the data packet of the non-real-time service; Step S3.2.4: When the data packet reaches the maximum framing length, the data packet is encapsulated in the MPDU packet area, and the data packets are combined into a data frame for transmission; Step S3.2.5: Calculate the average framing time, frame multiplexing efficiency and average packet delay of the non-time-sensitive service queue and output them.

6. The advanced on-orbit system intelligent framing method based on proximal strategy optimization according to claim 5, characterized in that: The network includes an agent, a reward, a buffer, and an environment. The agent has a strategy network and an evaluation network. The network parameters include the average packet arrival rate λ, the number of iterations G, the proportional coefficient of the decay rate ρ, the strategy network update period Tc, and the discount factor , clipping parameter c, number of self-update strategies nupdate, strategy network learning rate τ, and number of time slots sampled in one training iteration N.

7. The method for intelligent framing of an advanced on-orbit system based on proximal strategy optimization according to claim 6, characterized in that: The specific steps of the optimal framing time threshold algorithm described in step S3.1.3 are as follows: Step S3.1.3.1: Initialize network parameters θ and buffer zones; Step S3.1.3.2: Perform training iterations, dividing the training cycle into several consecutive time slots; t , build environment status s t , the set of environmental states for all time slots is represented as , time slot t Environmental status s t The definition is as follows: ； in, R s represents the average arrival rate of the queue in the time slot, L s represents the average queue length in the time slot, T f Indicates the average framing time of data in a time slot, L f Indicates the average effective frame length of the data frame in the time slot; Step S3.1.3.3: Agent receives environment state s t Input, according to the current environment status s t Based on the transition probability Generate Action a t , Indicates the current strategy and action a t Corresponding to the framing time threshold, save the action a t and the corresponding action probabilities , Indicates the old strategy; Step S3.1.3.4: Environment performs actions a t , with action a t The corresponding framing time threshold is used to perform framing operations and update the environment state; at the same time, a reward value is generated according to the reward function. r t , the reward function is as follows: ； In the formula, Indicates the environmental status s t Take action a t The reward function is L f is the average effective frame length of the data frame in the time slot, T f Indicates the average framing time of data in a time slot, T h is the framing time threshold, α and β are the adjustable weights of the average effective frame length and the average framing time, respectively, and , and δ Indicates the weight of the framing delay, representing the minimum and maximum allowed range; After the action is executed, the environment state of the next time slot is entered s t+1 , and pass it to the agent; Step S3.1.3.5: Constructing time slots t Environmental samples , save the environment sample to the buffer; Step S3.1.3.6: When the preset update time slot threshold is reached, the agent extracts environmental samples from the buffer and calculates the objective function as follows: ； in, represents the objective function, Indicates time slot t The expected value of Is the current strategy With the old strategy The ratio of is the joint advantage function estimated using generalized advantage, It is a hyperparameter in the strategy update process, which is used to limit the magnitude of the strategy update; The updated loss function is as follows: ； In the formula, loss represents the loss function, r i Indicates the current action a i The reward value, The current state of the environment calculated by the policy network s i of Q Value function, To evaluate the next environment state calculated by the network Q Value function, is the discount factor, a The next state s i+1 All possible actions, Is used for target Q Parameters of the target network for value calculation; The mean square error is calculated based on the following formula and the evaluation network is updated: ； In the formula, Is the loss function of the evaluation network, used to update the network parameters θ , G t From the time slot t The cumulative reward at the beginning, represents the state of the environment estimated by the value function s t The expected return, N is the number of time slots sampled in one training iteration; From time slot t Starting cumulative rewards G t As follows: ； In the formula, R t Indicates time slot t Instant rewards; When the action space is sufficient and the time slot is equal to the action standard deviation decay frequency, the action standard deviation is updated based on the following formula: ； in, σ t It is t The action standard deviation after the time slot update is: σ t+1 It is t +1 action standard deviation after time slot update, is the decay rate, ρ is the proportionality coefficient of the decay rate, σ min is the minimum action standard deviation; Next, the cumulative reward is calculated and saved to the buffer; Step S3.1.3.7: If the amount of data stored in the buffer exceeds the preset capacity, the buffer is cleared after the policy network parameters are updated; Step S3.1.3.8: After each round of training, update the environment status and prepare to enter the next round of training, and output the current framing time threshold value as the optimal framing time threshold value.

8. The advanced on-orbit system intelligent framing method based on proximal strategy optimization according to claim 1, characterized in that: For time-sensitive service queues, set the maximum number of encapsulated data packets in the MPDU packet area to M , the average packet arrival rate is λ , the framing time of the time-sensitive service queue is t sw , t is a time slot, then in the interval Inside, t sw The probability density function of for: ； Frame time t sw Framing time threshold The probability is: ； Average framing time of time-sensitive service queues As follows: ； Frame multiplexing efficiency of time-sensitive service queues As follows: ； In the formula, Indicates that within the time limit T The framing time of the time-sensitive service queue t sw Less than T The probability of l s is the packet length, l ms is the length of the MPDU packet area; Average packet delay of time-sensitive service queue As follows: ； In the formula, T d is the packet delay of the time-sensitive service queue, for Arrive within time n The probability of a packet, for Arrive within time M + x The probability of a packet.

9. The method for intelligent framing of an advanced on-orbit system based on proximal strategy optimization according to claim 1, characterized in that: For non-time-sensitive service queues, set S w is the framing time of the non-time-sensitive service queue, λ is the average packet arrival rate, M The maximum number of data packets encapsulated in the MPDU packet area. t is the time slot, then the average framing time of the non-time-sensitive service queue is As follows: ； Average packet delay of non-time-sensitive service queues for: ； In the formula, T τ is the packet delay of the non-time-sensitive service queue, i is the number of arriving packets.

Images