Satellite transmission congestion control method based on multi-agent deep reinforcement learning

Abstract

The application relates to a satellite transmission congestion control method based on multi-agent deep reinforcement learning, relates to the technical fields of artificial intelligence, satellite data transmission and information center network, and solves the problems that the satellite data transmission congestion control of the existing information center network architecture is low in intelligence, poor in adaptability and unable to change the congestion strategy from a global perspective, and the like, and improves the deficiency of the information center network congestion control in the present satellite data transmission. A FIB and PIT based on geographical position are proposed, time, geographical position longitude and latitude and other important time and space attributes in content cache preservation are saved, when a certain link of a satellite is disconnected (not all links are disconnected) and the cache content is affected, according to the predictable characteristics of the satellite cycle, the same cache content in other paths is calculated to be reacquired, so that the content acquisition failure of users (consumers) caused by link disconnection is avoided, and therefore the content acquisition method of the users (consumers) is expanded.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence, satellite data transmission and information center networks, and specifically to a multi-agent deep reinforcement learning (MADDPG) congestion control method for solving the congestion control problem in software-defined information-center networking (SDN) for satellite data transmission. Background Technology

[0002] With the reduction in satellite launch costs and the continuous improvement of air-to-ground network protocols, satellite-based applications are becoming increasingly diverse, such as satellite monitoring of coastlines and weather changes. In these applications, satellites generate massive amounts of data, but due to limitations in transmission technology and the distribution of ground stations, the data cannot be downloaded to the ground control center in a timely manner, affecting users' further data analysis operations such as feature extraction. Content-centric Information Centric Networking (ICN) represents a major future direction for network development. Its decoupling of physical location from content and strong mobility support make it particularly suitable for downloading massive amounts of satellite data. However, satellite links exhibit periodic connectivity and disconnection, rendering ICN's native congestion control ineffective for satellite data transmission. Furthermore, when a satellite link is disconnected, the paths stored in the ICN's PendingInterest Table (PIT) and Forwarding Information Base (FIB) become invalid (due to reasons such as interest packets needing to return along the same path but the link being interrupted, or consumer-cached link interruptions). This exacerbates congestion, preventing ground users (i.e., consumers) from accessing content and ultimately leading to transmission failure. Therefore, an intelligent, adaptive congestion control mechanism must be implemented based on the characteristics of satellite networks.

[0003] Currently, there is limited literature on congestion control in Information Center Networks (ICNs) for satellite transmission. Other dynamic networks can be used as a reference. Jessica McCarthy et al. addressed the difficulty of congestion control in ICNs within the dynamic network environment of vehicular networks by proposing priority numbering based on interest packets and link QoS. This reduces content retrieval time and avoids congestion affecting transmission. However, this mechanism does not analyze the entire congestion problem from a global perspective, making it unsuitable for large-scale satellite networks. Dehao Lan et al. introduced artificial intelligence reinforcement learning to achieve congestion control, but their research, conducted in conventional networks, did not consider the impact of dynamic networks on congestion and is also unsuitable for satellite networks. Xu Yaping et al. addressed the weakness of compressed storage in ICNs by redesigning the PIT and FIB storage structure tables, significantly reducing storage consumption during transmission and accelerating data table construction. Therefore, modifying the PIT and FIB structure tables is beneficial for data transmission in environments with dynamic topology changes.

[0004] In satellite data transmission, an ideal congestion control method should possess the characteristics of minimal data redundancy, data transmission adaptability to path load (performance), and intelligent adaptive congestion control from a global network perspective. However, existing congestion control technologies suffer from poor adaptability and low intelligence, and the current state can only meet the requirement of minimal data redundancy. Therefore, there is an urgent need to provide a highly intelligent and adaptive satellite transmission congestion control method. Summary of the Invention

[0005] To address the problems of low intelligence, poor adaptability, and inability to change congestion strategies from a global perspective in existing information center network architectures for satellite data transmission congestion control, this invention provides a congestion control method based on multi-agent deep reinforcement learning.

[0006] A congestion control method based on multi-agent deep reinforcement learning is implemented through the following steps:

[0007] Step 1: Initialize the SDN controller and the multi-agent deep reinforcement learning model;

[0008] Step 2: The consumer sends an interest packet to the content provider via the switch and obtains the content of the interest packet; specifically:

[0009] Determine if the interest packet speed, i.e. the instantaneous data transmission rate RN, is close to the maximum supported rate PR of the link. If not, the consumer reads and updates the contents of the PIT and FIB tables based on the geographical location to obtain the path. If the content is successfully obtained, the current transmission ends. If yes, start the congestion control mechanism and execute step three.

[0010] Step 3: Calculate and generate a congestion control mechanism using the multi-agent deep reinforcement learning model; that is: based on the network state information obtained by the SDN controller, randomly extract training data from the experience replay pool, calculate the loss function, jointly train all agents, and finally generate the action with the largest reward value at a certain moment, i.e., the congestion control action.

[0011] Step 4: The consumer performs the congestion control action, evaluates the consumer action, and updates the experience replay pool.

[0012] The beneficial effects of this invention are:

[0013] The method described in this invention expands the ways to obtain content. This invention addresses the shortcomings of current information center network congestion control in satellite data transmission. It proposes a geographic location-based FIB and PIT to store important temporal and spatial attributes such as time, longitude, and latitude of the content cached at that time. When a satellite link is disconnected (not all links are disconnected), affecting the retrieval of cached content, based on the predictability of satellite cycles, the same cached content is re-retrieved from other paths through calculation, thereby avoiding user (consumer) content retrieval failures due to link interruptions. Therefore, this invention expands the methods for users (consumers) to obtain content.

[0014] The method described in this invention enables intelligent congestion control, achieving highly adaptive congestion control. This invention utilizes an SDN controller to acquire global information about the entire satellite network: satellite topology, remaining bandwidth, hosts, cache nodes, etc., as parameters in multi-agent deep reinforcement learning. Each agent represents a deterministic policy gradient, and the consumer's content acquisition process is mapped to a Markov decision process. Through deep reinforcement learning, the congestion window for consumers sending interest packets is adjusted to improve throughput. Agents can learn from the experiences of other agents, resulting in faster convergence; they can also learn from the failures of other agents, making the program more robust. Attached Figure Description

[0015] Figure 1 This is a system architecture diagram of the satellite transmission congestion control method based on multi-agent deep reinforcement learning described in this invention.

[0016] Figure 2 This is a flowchart of the satellite transmission congestion control method based on multi-agent deep reinforcement learning described in this invention.

[0017] Figure 3 This is a comparison chart of throughput in the first 100 seconds for non-ICN and non-reinforcement learning within ICN using the method described in this invention. Detailed Implementation

[0018] Combination Figures 1 to 3This embodiment describes a satellite transmission congestion control method based on multi-agent deep reinforcement learning. This method achieves this through methods such as... Figure 1 The system implementation shown includes the following entities: Consumer, Provide User, SDN controller (with deep reinforcement learning congestion control module installed), and ICN switch (containing a location-based Pending Interest Table (PIT) and Forwarding Information Base (FIB)).

[0019] Consumer: Also known as user, who obtains content from content providers, i.e., needs to download data.

[0020] Content Provider (Provide User): Can provide content to consumers.

[0021] SDN controller (with deep reinforcement learning congestion control module installed): can obtain network node status information, perform congestion control calculations, and send flow tables to switches, etc.

[0022] ICN switches (including location-based PIT and FIB, Content Store): These switches function as switches in SDN networks and as caches and routes in data center networks. The location-based PIT and FIB are redesigned and updated based on the original data center network PIT and FIB, incorporating important temporal and spatial attributes such as content cache timestamps and geographical location longitude and latitude. Tables 1 and 2 show the location-based PIT and FIB tables, respectively.

[0023] Table 1

[0024] Primary key 13 Content primary key C242 Cache timestamp 1663469240 longitude 125.35 latitude 43.88 Cache node collection (W4, W6, W25) path / home / bj / 02 / x.lo003 ... ...

[0025] Table 2

[0026] Primary key 17 Cache timestamp 1663469297 longitude 115.35 latitude 93.88 path / home / jl / 09 / x.334 ... ...

[0027] This implementation includes establishing a multi-agent deep reinforcement learning model, consumers sending interest packet requests, calculating congestion control strategies, and consumers implementing congestion control (evaluating consumer actions and updating the controller's experience replay pool). The specific methods are as follows:

[0028] S1. Initialize the SDN controller and the multi-agent deep reinforcement learning model;

[0029] In this embodiment, the SDN controller is initialized. The SDN controller collects node status and link bandwidth to obtain all status information in the overall network. Specifically, the following status information is obtained through the LLDP (Link Layer Discovery Protocol):

[0030] Network topology: The link connection topology of the entire network, including hosts, switches with ICN functions, etc.

[0031] Cache distribution: Which switch a specific piece of content is located on.

[0032] Remaining bandwidth: The remaining available bandwidth on a certain link.

[0033] Location-based PIT and FIB tables: Retrieve detailed information about location-based PIT and FIB tables, such as cache timestamps and longitude and latitude information of the geographical location, as shown in Table 1.

[0034] In this embodiment, the multi-agent deep reinforcement learning model is as follows:

[0035] Each consumer is represented by an agent, denoted as C, and the total number of such agents is n. The i-th agent is denoted as Ci. i ,i∈[1,n]. The type of multi-agent deep reinforcement learning is: Mixed Cooperative and Competitive. When a consumer acquires content, the agents cooperate (the goal of all agents is to ensure the consumer successfully acquires the content); when multiple consumers acquire content, the agents compete (the agents compete for bandwidth resources). The entire deep reinforcement learning solution to the congestion control problem can be represented by a tuple (n,S,C,R,P), which is a Markov game process. The goal of deep reinforcement learning is for each agent to maximize its own reward (and maximize its total reward) through learning a policy, as detailed below.

[0036] The sum of all agent observations O = [o 1 ,o 2 ,o 3 ,…,o i ,…,o n ], where the observation value o of a certain agent belongs to O;

[0037] The sum of all agent states is S, and the state of an agent is denoted as s ∈ S. The total state of the deep reinforcement learning process is: S = [s...]. 1 ,s 2 ,s 3 ,…,s i ,…,sn ];

[0038] For action A, in order to make the action adjustment smooth, this invention proposes an adjustment base cwnd as shown in formula (1). adjust-value Store the size of the congestion window for the first 512 times in an array, and then calculate its average value, i.e.:

[0039]

[0040] Wherein, the instantaneous value of the congestion window is cwnd, and the congestion window adjustment base is cwnd. adjust-value Ultimately, the congestion window value needs to be adjusted, denoted as CWND.

[0041] When action a increases the congestion window, the possible actions are: CWND = CWND + cwnd adjust-value CWND = CWND + 5 × cwnd adjust-value CWND = CWND + 10 × Cwnd adjust-value There are 3 types of movements in total;

[0042] When action a reduces the congestion window, the possible actions are: CWND = CWND - 0.75 × cwnd adjust-value CWND = CWND - 0.5 × cwnd adjust-value CWND = CWND - 0.15 × cwnd adjust-value There are 3 types of movements in total;

[0043] When action a maintains the congestion window, there is only one action that can be executed: CWND = CWND. Since there are multiple agents, we have: A = [a...] 1 a 2 a 3 , ..., a i , ..., a n ], which is the sum of the actions of all intelligent agents.

[0044] Reward function R, R = [r 1 r 2 r 3 , ...r i ,...,r n [ ] is the sum of the reward values ​​of all agents, and the specific reward value will vary depending on the environment.

[0045] Let the environmental state transition probability be P, then P(a′|s, a) 1 a 2 a 3 , ..., a i , ..., a n ,)=P(S=s,C 1 =a1 C 2 =a 2 C 3 =a 3 C n =a n a′ represents the environmental state transition probability, which refers to the probability of performing action a from state s. i Then, the probability of obtaining a benefit.

[0046] Maximum supported link rate PR: PR = (BB) used ) / Δt, where B is the total bandwidth, B used The remaining bandwidth is the ratio of bandwidth used to time, which can be calculated by the SDN controller. When the instantaneous data transmission rate RN is close to (generally when RN receives PR*0.7) the maximum supported rate PR of the link, congestion may occur, and the SDN controller should intervene in a timely manner.

[0047] The SDN controller initiates the Actor and Critic networks, whose parameters are obtained from the link and transmitted by the SDN controller through the LLDP protocol.

[0048] S2. Consumer Requests Content: The consumer sends an interest package to content provider m via the handover station and obtains the content of the interest package. At this time, the agents are in the exploration phase, and the sum of the observations of all agents is 0; multi-agent deep reinforcement learning model is used for multi-agent deep reinforcement learning;

[0049] S2-1. Determine if the interest packet speed, i.e., the instantaneous data transmission rate RN, is close to the maximum supported rate PR of the link. If not, the cache hit is successful, and the consumer obtains the content of the PIT and FIB based on the geographical location, and proceeds to step S2-2. If yes, the cache hit fails (it may be the first time the content is obtained or the satellite link is disconnected). The content provider forwards the interest packet to the SDN controller. The SDN controller relocates other available cached content based on the consumer's geographical location and request time, and proceeds to step S3.

[0050] S2-2. Determine whether the PIT and FIB content based on the geographical location was successfully obtained. If not, the switch updates the timestamp, longitude, and latitude of the PIT and FIB based on the geographical location: update the time and space attributes such as the timestamp, longitude, and latitude of the PIT and FIB based on the geographical location, delete the invalid cached content path, and execute step S3; if yes, obtain and update the path based on the timestamp, longitude, and latitude of the PIT or FIB based on the geographical location, and the current transmission ends.

[0051] S3, generating congestion control mechanisms;

[0052] A congestion control mechanism is generated using a deep reinforcement learning model: The deep reinforcement learning module is activated, and based on the network state information obtained from the SDN controller, several training data points are randomly selected from the experience replay pool. The loss function is calculated, all agents are jointly trained, and the training process is scored and evaluated to generate the action that maximizes the reward at a given moment. The specific process is as follows:

[0053] S3-1. Calculate the reward r of each agent and the observation value o of a certain agent, and add and update the experience replay pool D.

[0054] S3-2. Obtain training data from the experience replay pool D. Each agent can be viewed as a DDPG (Deep Deterministic Policy Gradient), whose gradient formula is:

[0055]

[0056] The goal of formula (2) is to continuously explore an optimal strategy and maximize the expected return E,o of this strategy in the environment. n The first n consecutive strategies are denoted as μ n (o n ), J(μ n ) represents a continuous policy μ n The expected return The gradient formula for the decision parameter (weight value in a neural network) θ is given. This represents the gradient formula for the nth agent's action 'a', where D is the experience replay pool, storing data (o,…,o). n ,a 1 ,…,a n E O～D Let a represent the expected reward of a sample observation O taken from the experience replay pool, and let a represent the action that has adopted a certain strategy. n , It is a decentralized value function.

[0057] S3-3. According to formula (1), train the Actor and Critic networks of each agent in a centralized manner.

[0058] S3-4. Determine if the reward value is maximum. If yes, output it to step 4; otherwise, return to step 3 to continue training.

[0059] S4. Consumers implement congestion control actions (i.e., actions with the highest reward): Consumers execute actions with the highest reward, i.e., maximizing, reducing, or maintaining the sending window. Consumer actions are evaluated (scored by the Critic network in the multi-agent deep reinforcement learning model based on actions, states, etc.; a positive score rewards the action, while a negative score penalizes it) and the experience replay pool is updated.

[0060] S4-1, The agent executes the congestion mechanism action.

[0061] S4-2, End the traversal.

[0062] S4-3, Update the replay experience pool D.

[0063] Step 4-4: Update the agent's own target Actor and Critic networks. The agent remains in the exploration phase.

[0064] The method described in this embodiment was applied to a software-defined information center network composed of STK simulating a low-Earth orbit satellite Iridium constellation, with RYU as the controller and Mini-NDN, etc. After extensive experiments and simulations, the throughput was increased by approximately 7%. Figure 3 As shown, the method described in this embodiment is feasible and has significant effects.

[0065] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0066] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these all fall within the protection scope of the present invention. Therefore, the protection scope of this invention patent should be determined by the appended claims.

Claims

1. A satellite transmission congestion control method based on multi-agent deep reinforcement learning, characterized by: This method is implemented by the following steps: Step 1: Initialize the SDN controller and the multi-agent deep reinforcement learning model; The initialization of the SDN controller and the acquisition of all state information in the network specifically involves obtaining the network topology, cache distribution, remaining bandwidth, and PIT and FIB tables based on geographical location through the LLDP protocol. Step 2: The consumer sends an interest packet to the content provider via the switch and obtains the content of the interest packet; at this time, the agent is in the exploration phase, and the sum of the observations of all agents is 0; specifically: Step 21: Determine the packet speed, i.e., the instantaneous data transmission rate. Is the maximum supported rate of the adjacent link? If not, the cache hit is successful, and the consumer retrieves the contents of the PIT and FIB tables based on the geographic location and executes step two. If so, the cache hit fails, and the content provider forwards the interest packet to the SDN controller. The SDN controller then relocates other available cached content based on the consumer's geographic location and request time; proceed to step three. Step 22: Determine whether the PIT and FIB tables for obtaining the geographic location were successfully obtained. If not, the switch updates the time and spatial attributes based on the PIT and FIB timestamps, longitude and latitude, deletes invalid cached content paths, and proceeds to Step 3. If yes, the switch obtains and updates the path based on the PIT or FIB timestamps, longitude and latitude, and the current transmission ends. Step 3: Calculate and generate a congestion control mechanism using the multi-agent deep reinforcement learning model. That is: based on the network state information obtained by the SDN controller, training data is randomly extracted from the experience replay pool, the loss function is calculated, and all agents are jointly trained to finally generate the action with the maximum reward value at a certain moment, namely: the congestion control action. Step 4: The consumer performs the congestion control action, evaluates the consumer action, and updates the experience replay pool.

2. The satellite transmission congestion control method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: In step one, in the multi-agent deep reinforcement learning model: The sum of all observations from all agents is ; The sum of the states of all agents is: , ; The sum of all actions of the agents is , ; Set congestion window adjustment base It can be expressed as follows: Among them, the instantaneous value of the congestion window is The congestion window adjustment base is Ultimately, the congestion window value needs to be adjusted. ; When the sum of the actions of all intelligent agents The action of a certain intelligent agent When increasing the congestion window, three actions are performed: ； ； ； When the sum of the actions of all intelligent agents The action of a certain intelligent agent When reducing the congestion window, three actions are performed: ； ； ； When the sum of the actions of all intelligent agents The action of a certain intelligent agent When maintaining the congestion window, one action is to perform: ； The sum of the reward functions of the agents is , ; The probability of environmental state transition is :but ; The state transition probability of an environment refers to the probability of transitioning from a given state to a given state. Execute action Then, the probability of obtaining a profit; Maximum supported link rate : The instantaneous data transmission rate is calculated by the SDN controller. Maximum supported rate of adjacent links If this occurs, congestion will be triggered, and the SDN controller will execute the congestion control mechanism.

3. The satellite transmission congestion control method based on multi-agent deep reinforcement learning according to claim 1, characterized in that: The specific process of step three is as follows: Step 31: Calculate the reward value for each agent. Compared with observed values Added and updated to the experience replay pool middle; Step 3.2: From the experience replay pool Obtain training data from the middle; The gradient formula for each agent is: In the formula, A continuous strategy is denoted as , For continuous strategies The expected return Decision parameters The gradient formula, For the first Actions of an agent The gradient formula, For the experience replay pool, store data. , Represented as sample observations taken from the experience replay pool D The expected return, the action that has adopted a certain strategy is denoted as , A decentralized value function; Step 33: Based on the gradient formula of the agent, train the Actor and Critic networks of each agent in a centralized manner; Steps 3 and 4: Determine if the reward value is maximum. If yes, execute congestion control actions; otherwise, return to step 2.

Images