An asynchronous scheduling method for TSN-5G train communication network based on multi-agent reinforcement learning

Abstract

This invention provides an asynchronous scheduling method for TSN-5G train communication networks based on multi-agent reinforcement learning. The method includes: constructing a Markov model using a graph neural network based on the topology of a heterogeneous TSN-5G train scenario; defining the state space, action space, and reward function of the Markov model; inputting the data flow information to be processed in the heterogeneous TSN-5G network into a multi-agent reinforcement learning asynchronous scheduling algorithm; treating each base station and switch as an independent agent; updating the state space and action space of the Markov model through iterative computation; and outputting a scheduling scheme for the data flow to be processed. This invention proposes a comprehensive scheme that combines the utilization of wired and wireless channels and end-to-end jitter performance in TSN-5G to optimize data transmission, thereby achieving optimal control of end-to-end latency and jitter of the entire train network.

Description

Technical Field

[0001] This invention relates to the field of data stream transmission technology, and in particular to an asynchronous scheduling method for TSN (Time-Sensitive Networking)-5G train communication network based on multi-agent reinforcement learning. Background Technology

[0002] At the ingress port, the switch identifies and classifies frames based on identifiers and other information, resulting in flows of different priorities: TT (Time-Trigger), AVB (Audio / Video), and BE (Best Effort). Flow shaping limits the flow rate. Flow scheduling and preemption determine the flow scheduling order. Finally, the switch forwards the traffic at the egress port to the target device or the next switch.

[0003] The purpose of traffic shaping is to alleviate network congestion and reduce packet loss rate. This is achieved by limiting and buffering bursts of traffic, controlling the flow to be sent at a relatively uniform rate. Currently, common shaping techniques include Credit-Based Shaping (CBS) in clock-synchronized scenarios and Asynchronous Traffic Shaping (ATS) in asynchronous scenarios.

[0004] Flow scheduling ensures the orderliness of flow forwarding and reduces network congestion. The main indicators to consider are latency and jitter.

[0005] (1) Time-Aware Shaper (TAS)

[0006] IEEE 802.1Qbv defines the Transmission Frame Allocation System (TAS) and gated list technology. TAS allocates different transmission time slots to each transmission window, reducing interference between different data streams. A gating structure is used to control the transmission of data frames. The gating list records the opening and closing status of all queue gates at a given time.

[0007] (2) Cyclic Queuing and Forwarding (CQF)

[0008] IEEE 802.1Qch introduced the CQF (Concurrent Frame Flow) mechanism, allowing switches to synchronously transmit frames in a cyclic manner. The CQF process is briefly described below: Assume there are two transmission queues and a cycle. In even-numbered cycles, queue 1 accumulates frames from the switch's receive port, while queue 2 transmits frames accumulated in odd-numbered cycles. In odd-numbered cycles, queue 1 sends frames accumulated in even-numbered cycles, and queue 2 receives frames from the switch's receive port. With a properly configured cycle, any latency-sensitive data stream can be scheduled and transmitted within the cycle.

[0009] Currently, existing stream scheduling technologies include scheduling algorithms for synchronous and asynchronous scenarios, depending on whether clock synchronization is required.

[0010] Scheduling algorithms in clock synchronization scenarios

[0011] Research on scheduling algorithms in synchronous scenarios includes those in TSN and TSN-5G converged scenarios.

[0012] (1) Prediction of switch queue length

[0013] The length of the queue in each exchange is predicted using an enhanced Long Short-Term Memory (LSTM) network. LSTM is a variant of Recurrent Neural Network (RNN) and can effectively handle temporal and spatial sequence data. Compared to RNN, LSTM can selectively store information through gated transposition.

[0014] In this approach, the model uses network topology, traffic type, and queue length as input, and queue length as output to perform a prediction task. Specifically, it predicts the future number of queued frames based on the current network condition and the number of queued frames at the switches. This serves as an indicator of network congestion. Routing strategies are then adjusted based on the prediction results to achieve dynamic routing. However, using queue length as a representative of network congestion is too simplistic and cannot maximize the reduction of end-to-end latency and jitter.

[0015] (2) TSN-5G Particle Swarm Reinforcement Learning Algorithm

[0016] An integrated industrial TSN-5G network architecture is proposed. A Double Q-Learning Hierarchical Particular Swarm Optimization (DQHPSO) algorithm is also proposed. This algorithm simulates the search behavior of a swarm of particles in a multidimensional space, where each particle represents a potential solution and updates its position and velocity based on the historical best positions of both the individual particle and the swarm.

[0017] With scheduling success rate as the optimization objective, scheduling is considered successful if each switch can transmit data within the transmission window, thus satisfying the CQF cycle. This ensures the final latency and jitter metrics.

[0018] (3) TSN-5G joint scheduling takes into account 5G channels

[0019] This method considers 5G time slot utilization and the impact of SINR (signal-to-noise ratio), i.e., signal quality, on data transmission, and uses reinforcement learning algorithms to improve signal utilization. 5G uses millisecond-level time slots; this method introduces micro-sized time slots to improve channel utilization.

[0020] The drawbacks of the existing flow scheduling techniques mentioned above include: in existing joint scheduling methods for TSN-5G synchronous or asynchronous networks, some studies focus on the impact of jitter, while others emphasize improving channel utilization to optimize latency performance. However, considering jitter alone may lead to a significant increase in latency, while focusing solely on channel utilization may cause large data frame transmission jitter. In practical train communication scenarios, data transmission needs to simultaneously meet the dual requirements of low latency and low jitter to ensure communication efficiency and determinism.

[0021] Existing reinforcement learning methods for TSN-5G heterogeneous scenarios are mostly based on single-agent architectures. However, in such scenarios, each base station and each switch acts as an independent scheduling unit, facing different scheduling tasks. Furthermore, there are cooperative and restrictive relationships between base stations, between base stations and switches, and between switches. Therefore, this invention designs a scheduling method based on multi-agent reinforcement learning, applicable to TSN-5G heterogeneous network scenarios, aiming to build a more comprehensive and cooperative scheduling model, thereby improving the overall system performance.

[0022] Existing reinforcement learning methods in heterogeneous TSN-5G scenarios often only accelerate the convergence process of optimization algorithms by considering network topology relevance, without fully considering the potential impact of network topology on scheduling results. Differences in network topology directly affect network traffic load, scheduling decisions, and bandwidth resource utilization. To address this issue, this invention introduces graph neural networks to construct a more accurate network topology model, thereby improving the performance of scheduling methods under different topology structures. Summary of the Invention

[0023] The embodiments of the present invention provide an asynchronous scheduling method for TSN-5G train communication networks based on multi-agent reinforcement learning, so as to achieve optimal control of end-to-end latency and jitter of the entire vehicle network.

[0024] To achieve the above objectives, the present invention adopts the following technical solution.

[0025] An asynchronous scheduling method for TSN-5G train communication network based on multi-agent reinforcement learning includes:

[0026] Obtain the topology of the heterogeneous TSN-5G scenario of the train;

[0027] Based on the topology of the TSN-5G heterogeneous scenario of the train, a Markov model is constructed using a graph neural network, and the state space, action space and reward function of the Markov model are defined.

[0028] The data stream information to be processed in the TSN-5G heterogeneous network is input into the Train TSN-5G Multi-Agent Reinforcement Learning Asynchronous Scheduling Algorithm. The Train TSN-5G Multi-Agent Reinforcement Learning Asynchronous Scheduling Algorithm treats each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs the scheduling scheme of the data stream to be processed.

[0029] Preferably, the acquisition of the topology of the train TSN-5G heterogeneous scenario includes:

[0030] The train TSN-5G heterogeneous scenario includes onboard 5G terminal equipment (UE), 5G base station (BS), user function plane (UPF), TSN switch (SW), and driver control console (Server). A centralized network configuration with a CNC server for overall control is used. There are N UEs in total, denoted as UE = {ue1, ue2, ..., ue...}. N There are M elements in total, denoted as B = {b1, b2, ..., b}. M There are S elements SW in total, denoted as SW = {sw1, sw2, ..., sw}. S The entire device comprises L data streams, and device U... i It will send a data stream L i Item, denoted as set up The number of data frames sent in the current data stream. The size of a single data frame in the data stream. For data stream sending period, This represents the average latency of the current data stream.

[0031] The BS status settings include the data frame wireless transmission time T. w Number of BS queued data frames Q i and channel usage U channel The status settings for SW include the data frame arrival time T. a Data frame compliance time T r Accumulated waiting time T for data frames wait Link utilization U of each port of the switch link .

[0032] Preferably, the topology based on the heterogeneous scenario of the train TSN-5G utilizes a graph neural network to construct a Markov model, defining the state space, action space, and reward function of the Markov model, including:

[0033] The Markov decision process is extended to the multi-agent environment of the heterogeneous scheduling scenario of train TSN-5G. The collaborative scheduling problem between SW and BS in TSN-5G is modeled as a Markov game model. In the Markov game model, each base station and switch is treated as an independent agent. Each agent calculates the reward function based on the perceived environmental state and selects different actions.

[0034] The six-tuple in a Markov game model is defined as follows (M+S, State, Action). M+S ,P,γ,Reward M+S In the system, M+S represents the number of agents, State represents the system state (either the joint state of the agents or a partially observable state of a single agent), Action represents the joint action space of the agents, P represents the state transition function (i.e., the probability distribution of the state value at the next moment given the current system state and the joint action space Action), and R represents the state transition function. i ,i∈{1,…,M+S} is the reward function of the i-th agent;

[0035] In the train TSN-5G multi-agent data frame scheduling environment, considering multiple switchers and base operators, the state space, action space, and reward function are defined as follows:

[0036] State Space: In the CNC server, each agent obtains overall load information and some of its own parameters. For agent B in the BS architecture... i The definition of the observed state at time t is shown in equation (3);

[0037]

[0038] Representing the BS intelligent agent B i The state is observed at time t. Representative in BS intelligent agent B i The wireless transmission time of the data frame arriving at time t. Representing the BS intelligent agent B i The length of the data frame queue at time t (data frames will queue in the BS agent due to network performance limitations). Representing the BS intelligent agent B i Wireless channel utilization at time t;

[0039] For SW agent sw i The definition of the observed state at time t is shown in equation (4);

[0040]

[0041] Representing the SW intelligent agent sw i The state is observed at time t. Representative in SW agent sw i The ATS parameter of the data frame arriving at time t—the arrival time. Representing the SW intelligent agent sw i The ATS parameter of the data frame arriving at time t—compliance time. Representing the SW intelligent agent sw i The cumulative waiting time of the data frame arriving at time t. Representing the SW intelligent agent sw i Wired link utilization at time t;

[0042] According to the definition and The global state space is obtained as shown in equation (5);

[0043]

[0044] S t This represents the global observation state at time t. The partially observable state space representing the agent. to It is a portion of the observable state space of each BS. to It is a partial observable state space of each SW;

[0045] Action Space: The action spaces of BS and SW are different. Each BS has the same actions, and each SW has the same actions. BS agent B... i With SW agent sw i The actions that can be selected at time t are: For B i ,action For wireless transmission frequency and latency, for SW i ,action Waiting time;

[0046] Reward function: Each agent uses a target function as its reward. The transmission delay of a single data frame is shown in equation (1), where T ij Representative device ue i Data stream sent Queuing time in BS For the BS to SW link transmission time, For the link transmission time between SW, The processing time of the data frame in the switch. The time it takes for a data frame to be sent in the switch. The queuing time of data frames in the switch;

[0047]

[0048] The objective function is set as shown in equation (2) with the goal of minimizing data frame transmission delay and minimizing data frame transmission jitter:

[0049]

[0050] Preferably, the step of inputting the data stream information to be processed in the TSN-5G heterogeneous network into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm, wherein the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm treats each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs a scheduling scheme for the data stream to be processed, includes:

[0051] In the MADGNN algorithm of TSN-5G collaborative scheduling, BS and SW are used as agents. Each BS and SW has an independent actor network and a critic network. The actor network and the critic network are deployed on the CNC server.

[0052] The Base Station (BS) obtains the current system's environmental state space through the Critic network. This environmental state space includes channel utilization, data frame queuing count, and data frame wireless transmission time. The BS inputs this environmental state space into its Actor network, which generates actions including data stream transmission frequency and latency. The SW obtains the current device's environmental state space through the Critic network. This environmental state space includes data frame arrival time, compliance time, cumulative latency, and SW link utilization. The SW inputs this environmental state space into its Actor network, which generates actions including data stream latency.

[0053] After BS and SW perform actions, the system provides a reward value. The system stores the current state, action, reward, and next state as a quadruple in the experience replay pool. When the number of samples in the replay pool reaches a threshold, the CNC server extracts samples from the replay pool and uses the quadruple data in the samples to train and update the Actor network and Critic network.

[0054] Preferably, the CNC server extracts samples from the playback pool and uses the quadruplet data in the samples to train and update the Actor network and the Critic network, including:

[0055] The CNC server extracts samples from the playback pool and transmits the quadruple data in the samples to the Actor network and the Critic network. The Actor network and the Critic network adopt a dual network structure, using both the current network and the target network.

[0056] The Critic network uses the quadruples data in the experience replay pool to calculate the evaluation value Q of the Actor network and performs gradient updates based on the TD error, as shown in Equation (6). The MSE loss function calculated using the TD time-series difference error is shown in Equation (7). The parameters are updated using gradient descent.

[0057]

[0058] In the above formula, The parameters of the Critic network, and their expected values. The quadruple data sampled from the experience replay pool includes the current state s, the action taken a, the immediate reward r after execution, the next state s′ after transition, and a flag d indicating whether it is a terminating state. The current Q-value function is... Estimate the value of taking action a in state s;

[0059] The Actor network uses the quadruples data from the experience replay pool to calculate the MSE loss function of the TD error, as shown in Equation (8), to calculate the loss function with respect to θ. i The gradient is used to update the parameters using gradient descent, as shown in equation (9);

[0060]

[0061] θ i The parameters of the Actor network, and the expected symbol. The expected value of a state drawn from the experience replay pool D is used to estimate the average effect across the entire state space, and the policy gradient. The loss function is expressed as a function of the policy parameter θ. i The derivative of the gradient is used to indicate the direction of parameter optimization; it is included in the gradient formula. The term represents the rate of change of the Q-value function relative to the policy parameters. This indicates that the Actor network is in a given state s. i The output action below;

[0062] The target network is updated using a soft update method, as shown in equation (10):

[0063]

[0064] This represents the parameters of the main Q network. Here, θ represents the parameters of the target Q-network, θ represents the parameters of the master policy network used to select actions, and θ′ represents the parameters of the target policy network. The parameter τ is an update coefficient, between 0 and 1, controlling the influence of the master network parameters on the target network parameters. In the update rule... Parameters of the main Q network and its original value The values ​​of τ and (1-τ) are mixed proportionally, and θ′ is obtained by mixing the parameters θ and their original values ​​θ′ of the main policy network proportionally.

[0065] As can be seen from the technical solutions provided by the embodiments of the present invention above, the present invention proposes a comprehensive solution that combines the utilization rate of wired and wireless channels and end-to-end jitter performance in TSN-5G to achieve the purpose of optimizing data transmission.

[0066] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0067] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0068] Figure 1 This is a schematic diagram of a train TSN-5G heterogeneous scenario provided by an embodiment of the present invention;

[0069] Figure 2 A flowchart illustrating an asynchronous scheduling method for TSN-5G train communication network based on multi-agent reinforcement learning, provided in an embodiment of the present invention;

[0070] Figure 3 A data frame transmission flowchart is provided as an embodiment of the present invention;

[0071] Figure 4 This diagram illustrates the implementation principle of a TSN-5G scheduling algorithm based on MADGNN (Multi-Agent Deep deterministic policy gradient with Graph Neural Network, multi-agent TSN-5G heterogeneous network joint scheduling) in a Markov game model provided by an embodiment of the present invention. Detailed Implementation

[0072] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0073] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0074] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0075] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0076] This invention uses 5G as a transparent bridge solution, used only for data transmission. It interfaces the TSN network with the 5G network, maintaining relative network independence within the overall system. Deterministic transmission of services is achieved through segmentation to improve end-to-end service delivery quality. The TSN network retains its traffic configuration markers; after remote transmission through the 5G network system, the 5G encapsulation is stripped, and upon entering the collaborative service system, deterministic transmission continues according to the TSN traffic scheduling type. The service system networks at both ends are TSN networks, while the 5G air interface, 5G bearer network, and 5G core network in between are 5G networks.

[0077] In train-mounted network communication scenarios, introducing 5G wireless technology reduces cabling and enhances system flexibility. However, wireless technology faces uncertainties in end-to-end data transmission due to complex environments and numerous devices. Therefore, TSN technology is used to enhance determinism. Existing synchronous methods suffer from issues such as clock asynchrony between TSN and the heterogeneous 5G network, as well as additional overhead. Existing asynchronous methods suffer from excessively long waiting times, leading to high latency, and do not consider the constraints between different switches or the impact of switch topology on scheduling results.

[0078] This invention proposes a joint asynchronous scheduling method for base stations and switches that considers the cooperative constraints between base stations, between base stations and switches, and between switches, as well as the impact of the overall network topology, thereby reducing data transmission latency and jitter. This alleviates the uncertainty in data transmission caused by the introduction of wireless transmission in industrial Ethernet. Here, base stations and switches are treated as two types of agents in a multi-agent environment.

[0079] This invention provides an embodiment of a train TSN-5G heterogeneous scenario, such as... Figure 1 As shown. The entire system includes 5G user equipment (UE), 5G base station (BS), UPF (User Function Plane, located in the 5G core network), TSN switch (SW), and driver console (Server). There are N 5G user equipment units in total, denoted as UE = {ue1, ue2, ..., ue...}. N},exist Figure 1 Taking the Traction Control Unit (TCU), Brake Control Unit (BCU), and Door Control Unit (DCU) as examples; there are a total of M units in BS, denoted as B = {b1, b2, ..., b...} M There are S elements SW in total, denoted as SW = {sw1, sw2, ..., sw}. S All devices comprise a total of L data streams, device U i It will send a data stream L i Item, denoted as In addition to the equipment described above, CNC (Centralized Network Configuration) equipment will be used for overall control, reducing the computational burden on the training and deployment models of switches and base stations. The number of data frames sent in the current data stream. The size of a single data frame in the data stream. For data stream sending period, This represents the average latency of the current data stream.

[0080] For the BS, state content such as the wireless transmission time T of a data frame can be defined. w This represents the transmission time of a data frame under conditions that may be affected by wireless channel quality interference, which is the main cause of transmission jitter in the entire transmission model; the number of BS queued data frames Q. i , represented as B i The number of currently queued data frames; channel usage U channel This indicates the current channel occupancy status, which facilitates further improvement of channel utilization through algorithms.

[0081] For SW, state content such as data frame arrival time T can be defined. a Because this is a heterogeneous scenario, the arrival time here is not the globally synchronized time, and will work in conjunction with the compliance time; data frame compliance time T r The compliance time is a parameter calculated by the ATS algorithm, and is related to the arrival time T. a The combined representation indicates the current queuing status of the queuing system, facilitating adjustments to the current SW load through model actions; the cumulative waiting time T for data frames. wait This parameter represents the waiting time of the data frame at the BS and the SW that has already passed through, indicating the time the data frame has waited through the joint scheduling model, in order to better predict and infer the actions that the SW will perform; the link utilization U of each port of the switch. link This represents the utilization rate of the outgoing port of the switch (SW), thus indicating the link utilization rate of different traffic paths. This facilitates the improvement of link utilization and the balancing of link utilization between different SWs through the scheduling model in the SW, preventing some devices from being overloaded while the utilization rate of other devices is too low.

[0082] The flowchart of an asynchronous scheduling method for TSN-5G train communication network based on multi-agent reinforcement learning provided in this embodiment of the invention is as follows: Figure 2 As shown, the processing steps include the following;

[0083] Step S10: Obtain the topology of the train TSN-5G heterogeneous scenario.

[0084] Step S20: Based on the topology of the TSN-5G heterogeneous scenario of the train, a Markov model is constructed using a graph neural network, and the state space, action space and reward function of the Markov model are defined;

[0085] Step S30: Input the data stream information to be processed in the above TSN-5G heterogeneous network into the Train TSN-5G Multi-Agent Reinforcement Learning Asynchronous Scheduling (MADGNN) algorithm. The MADGNN algorithm treats each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs the scheduling scheme of the data stream to be processed.

[0086] The data flow scheduling scheme is essentially the action generated in the Markov model of the reinforcement learning algorithm. The base station generates actions—transmission frequency and waiting time—by acquiring states such as channel utilization, data frame queue size, and data frame wireless transmission time. Regarding transmission frequency, 5G base stations have available time and frequency resources. This algorithm uses the base station to allocate frequency resources for data frame transmission, generating different frequency allocation schemes based on channel utilization and the number of data frames queued at the base station. As for the waiting time, also called the hold time, the base station generates the waiting time based on the aforementioned states, and the data frame will be held at the base station for a certain period.

[0087] Similarly, wait times are set in the switch (SW) device to track data frame arrival time, data frame compliance time, cumulative data frame wait time, and SW link utilization generation wait time. The inclusion of wait times in both the BS and SW settings ensures load balancing.

[0088] An embodiment of the present invention provides a data frame transmission process as follows: Figure 3 As shown. The data frame is generated by the UE, encapsulated into a 5G frame by the DSTT (Device-side TSN Translator) component, and then wirelessly transmitted to the 5G base station. The 5G base station executes a scheduling algorithm to improve link utilization. The 5G frame is decapsulated by the NWTT (Network-side TSN Translator) component in the core network. The data frame is transmitted to the TSN network and enters the SW. The SW executes the TSN-5G scheduling algorithm based on MADGNN, allowing the corresponding data frame to wait for a period of time to alleviate the jitter problem caused by wireless transmission. Finally, it will reach the target device after passing through multiple switches. The transmission of this data frame is completed. The BS and SW obtain different state spaces as shown in Equation (3) and Equation (4) respectively to make decisions. The decisions are all generation actions, and the scheduling algorithm is still executed.

[0089] Based on the above single data frame transmission process, the transmission delay of a single data frame can be obtained as shown in equation (1), where T ij Representative device ue i Data stream sent Queuing time in BS For the BS to SW link transmission time, For the link transmission time between SW, The processing time of the data frame in the switch. The time it takes for a data frame to be sent in the switch. The queuing time for data frames in the switch.

[0090]

[0091] The BS and SW will generate actions based on the system status. The BS will generate the radio transmission frequency and waiting time based on channel utilization, data frame queuing count, and data frame radio transmission time. The SW will generate the waiting time based on data frame arrival time, data frame compliance time, cumulative data frame waiting time, and SW link utilization.

[0092] This invention aims to improve network transmission performance by minimizing data frame transmission delay and jitter, and solves the multi-objective optimization problem of delay and jitter in the TSN-5G heterogeneous network of trains. The objective function is defined as shown in equation (2).

[0093]

[0094] Where λ and μ are hyperparameters, dynamically adjusted according to different training effects (different latency and jitter metrics). T ij Representative device ue i Data stream sent The sum of the times it takes to reach the destination device is the delay of the data stream. Represents ue i Device data stream The average delay. In the formula The jitter of all devices will be averaged.

[0095] This invention provides a Markov game model for a train TSN-5G communication scenario, comprising: in a heterogeneous scheduling scenario of train TSN-5G, each SW and BS acts as an agent, extending the Markov decision process to a multi-agent environment, and modeling the SW-BS collaborative scheduling problem in TSN-5G as a Markov game model. In the Markov game model, multiple agents (i.e., SW and BS) interact, with mutual constraints and cooperation existing between BS, between BS and SW, and among SWs themselves, resulting in system uncertainty. Each agent will choose different actions based on the perceived environmental state, influencing the generation of the next state. Due to the uncertainty of the system, the reward function that each agent needs to calculate depends not only on the reward generated by its own actions but also on the reward functions obtained from executing the actions of other agents, aiming to maximize long-term rewards. The entire system will balance the reward value distribution of each agent to maximize the system's overall benefit.

[0096] In Markov game models, a six-tuple is typically used for formal description. Substituting this into the scenario defined in this invention, the six-tuple is described as follows: (M+S, State, Action) M+S ,P,γ,Reward M+S In this system, M+S represents the number of agents, which in this scenario are the number of BS and SW; State represents the system state, generally referring to the joint state of multiple agents or a partial observable state of a single agent. In this scenario, due to the use of a CNC server, the joint state will be used as the overall input; Action represents the joint action space of the agents; P is the state transition function, which is the probability distribution of the state value at the next moment given the current system state and the joint action space Action; R i ,i∈{1,…,M+S} is the reward function of the i-th agent. Since the system objective will comprehensively consider the delay and jitter of data frame arrival, the reward function of each agent is consistent in this invention.

[0097] In the train TSN-5G multi-agent data frame scheduling environment, considering multiple switchers and base operators, the state space, action space, and reward function are defined as follows:

[0098] State Space: In the CNC server, each agent can obtain overall load information and some of its own parameters. For the BS agent B... i The state observed at time t is defined as shown in equation (3).

[0099]

[0100] Representing the BS intelligent agent B iThe state is observed at time t. Representative in BS intelligent agent B i The wireless transmission time of the data frame arriving at time t. Representing the BS intelligent agent B i The length of the data frame queue at time t (data frames will queue in the BS agent due to network performance limitations). Representing the BS intelligent agent B i Wireless channel utilization at time t.

[0101] For SW agent sw i The state observed at time t is defined as shown in equation (4).

[0102]

[0103] Representing the SW intelligent agent sw i The state is observed at time t. Representative in SW agent sw i The ATS parameter of the data frame arriving at time t—the arrival time. Representing the SW intelligent agent sw i The ATS parameter of the data frame arriving at time t—compliance time. Representing the SW intelligent agent sw i The cumulative waiting time of the data frame arriving at time t. Representing the SW intelligent agent sw i Wired link utilization at time t.

[0104] According to the definition and The global state space can be obtained as shown in equation (5).

[0105]

[0106] S t This represents the global observation state at time t. The partially observable state space representing the agent. to It is a portion of the observable state space of each BS. to It is a partial observable state space of each SW.

[0107] Action Space: The action spaces of BS and SW are different. Each BS has the same actions, and each SW has the same actions. BS agent B i With SW agent sw i The actions that can be selected at time t are: For B i,action This refers to the wireless transmission frequency and latency. For SW... i ,action For waiting time.

[0108] Reward function: Each agent uses a target function as its reward.

[0109] To reduce end-to-end latency and jitter in TSN-5G heterogeneous network data frame transmission, global state information, action space, and reward function are defined in a Markov game model. The implementation principle of a MADGNN-based TSN-5G scheduling algorithm in a Markov game model provided in this embodiment of the invention is as follows: Figure 4 As shown, each BS and SW is an agent, possessing an independent Actor network and Critic network. Since the system uses a CNC server, both the Actor and Critic networks are deployed on the CNC server, and network training and deployment are centrally executed within the CNC server. For multi-agent algorithms, there are centralized training and decentralized execution. Specifically, centralized training is achieved by obtaining global information through the Critic network to collaborate with the Actors. After training, each agent makes waiting time or frequency decisions through its independent Actor network.

[0110] The Actor network uses a hierarchical graph neural network, while the Critic network uses a global graph neural network. Agent B i or sw i The Actor policy network is denoted as That is, the Actor network can adjust the state g of the agent itself. i Make a decision i The Critic network calculates its Q-value and is used to train and update the Actor network. During training, the algorithm simultaneously learns the value function and policy function. The Critic, acting as the agent's value network, is denoted as...

[0111] Both the Actor and Critic networks employ a dual-network approach, using both the current network and the target network to avoid overestimation caused by using the same network when updating the Q-value. Agent B i or sw i The target Actor network is denoted as Used to determine the next action based on a strategy. The target Critic network is denoted as Used to calculate the target Q value.

[0112] Training data is obtained through the interaction between the agent and the environment. For example, in the data frame transmission process, the UE sends data, and at time t, the BS or SW receives the data frame and makes a waiting time decision or frequency decision based on the observed state. This constitutes the global state S and the action set A. The executed actions are fed back to the environment, resulting in a reward set R for each agent. The environment state changes from S at time t to S' at time t+1. This generates a quadruple (S, A, R, S'), which is placed in the experience backtesting pool. After accumulating a certain amount, it is used to train the Critic network and Actor network.

[0113] For the Critic network, the MSE loss function for both is constructed using the TD error (Temporal Difference Error) as shown in Equation (6), and the parameters are updated using gradient descent. As shown in equation (7).

[0114]

[0115] In the above formula, This represents the parameters of the Q-value network (i.e., the Critic network), which the network optimizes to estimate the value of state-action pairs. The formula includes the expected value of a set of data sampled from the experience replay pool. This includes the current state s, the action taken a, the immediate reward r after execution, the next state s′ after transition, and a flag d indicating whether it is a terminating state. Current Q-value function. Estimate the value of taking action a in state s. The objective Q-value is partly determined by the immediate reward r. i Discount factor γ and optimal Q-value estimation for the next state Composition. A discount factor controls the emphasis on future rewards; if the next state is a terminal state, future rewards are not included. The goal of the loss function is to minimize the mean squared error between the current Q-value and the target Q-value, thus gradually bringing the network's estimate closer to the true value. Gradient. This indicates the trend of the loss function with respect to the Q-value and network parameters. It allows you to adjust the parameters to minimize the loss.

[0116] For the Actor network, when calculating the forward propagation of each agent's own Actor, each agent only uses the local observed state as the input to the Actor network to obtain a deterministic action. The MSE loss function for calculating the TD error is shown in equation (8), and the loss function with respect to θ is calculated. i The gradient is used to update the parameters using gradient descent as shown in equation (9).

[0117]

[0118] θi This represents the parameters of the policy network (i.e., the Actor network), which are optimized by minimizing the loss function to improve policy performance. Expectation symbol. This represents the expectation of drawing states from the experience replay pool D, used to estimate the average effect across the entire state space. Similarly, It is the Q-value function, which estimates the value of taking action a in state s. Policy gradient The loss function is shown in relation to the policy parameter θ i The derivative of the gradient is used to indicate the direction of parameter optimization. It is included in the gradient formula. The term represents the rate of change of the Q-value function relative to the policy parameters, and also includes... Indicates the policy in a given state s i The output action is determined by these terms. These terms work together to compute the policy gradient, which is used to update the parameters θ of the policy network. i This optimizes the behavior of the intelligent agent.

[0119] The target network is updated using a soft update method, as shown in Equation (10).

[0120]

[0121] This represents the parameters of the main Q-network, which are frequently updated directly using the training data. θ represents the parameters of the target Q-network. The parameters of the target Q-network are updated slowly, primarily to provide stable target values ​​during training. θ represents the parameters of the master policy network, used for action selection, while θ′ represents the parameters of the target policy network, similar to the target Q-network, used to provide a more stable policy objective. The parameter τ is an update coefficient, between 0 and 1, controlling the degree of influence of the master network parameters on the target network parameters. In the update rule, Parameters of the main Q network and its original value The parameter θ is obtained by mixing τ and (1-τ) proportionally. Similarly, θ′ is obtained by mixing the parameter θ of the main policy network proportionally with its original value θ′. In this way, by making only small parameter updates each time, the target network can more stably approximate the performance of the main network.

[0122] The processing procedure of the TSN-5G heterogeneous collaborative scheduling MADGNN algorithm is shown in Algorithm 1. The algorithm initializes the agent set and the experience replay pool. Through multiple rounds of iterative training, each agent (BS or SW) observes the current environmental state at each time step t in each round and obtains an action through the Actor network, such as base station frequency allocation or data frame waiting time of the switch. After all agents have executed their actions, the system will provide a reward value and proceed to the next environmental state. The current state, action, reward, and next state are stored in the experience replay pool. When the experience replay pool reaches a certain number of samples, the network is updated. Based on samples randomly drawn from the replay pool, the parameters of the Actor and Critic networks are updated, and the target network is synchronized using a soft update method until the algorithm ends.

[0123]

[0124]

[0125] In the MADGNN algorithm for TSN-5G collaborative scheduling, each agent (including the base station (BS) and switch (SW)) uses an independent Actor network for decision-making. The base station generates actions—transmission frequency and waiting time—based on the current environmental state (such as channel utilization, data frame queuing count, and data frame wireless transmission time). The transmission frequency is allocated by the base station's frequency resources under given conditions to adapt to different channel utilization and queuing situations; the waiting time (or hold time) determines the time data frames are queued at the base station to ensure stable traffic flow.

[0126] Similarly, the switch generates corresponding latency based on the arrival time of data frames, compliance time, cumulative latency, and SW link utilization through the Actor network. These latency times are designed to maintain load balancing between the BS and SW, mitigating potential jitter issues during transmission. The switch schedules data flows according to the set latency times, ensuring a smooth rhythm of data frames across the entire transmission link.

[0127] Specifically, Actor networks and Critic networks are independent neural networks, such as... Figure 4 As shown. The Critic network uses a global network to extract global network topology features, with BS, UPF, and SW serving as nodes in the Critic network. The Actor network uses the local graph of the corresponding device to extract local network topology features, connecting BS or SW agents with adjacent connected agents as nodes in the Actor network.

[0128] The input layer of the BS's Actor network is also a state space, namely channel utilization, data frame queuing count, and data frame radio transmission time; the output layer is also an action space, with two actions: radio transmission frequency and waiting time. The input layer of the SW's Actor network consists of data frame arrival time, compliance time, cumulative waiting time, and SW link utilization; the output layer has only one action—waiting time, which is the waiting time for the data frame to be retained in the SW.

[0129] After the action is performed, the system will provide a reward value. This reward value is calculated based on indicators such as transmission delay and jitter. In other words, the system uses formula (2) to calculate the reward value to reflect the quality of the agent's behavior. The system enters the next environmental state. The next state can be directly input based on the actual state of the system at the next moment (in this model, the next moment of a certain agent BS or SW is defined as the state when the next data frame arrives at the agent).

[0130] The system stores a quadruple consisting of the current state, action, reward, and next state in the experience replay pool. When the number of samples in the replay pool reaches a threshold, the algorithm begins to extract samples from it for training the network.

[0131] The Critic network updates its Q-value by evaluating the quality of the agent's actions, thus aiding the training of the Actor network. The Critic network calculates the value of state-action pairs, providing the Actor network with a target to help the agent choose the optimal action policy. By using quadruples of data from the experience replay pool, the Critic network performs gradient updates based on TD error, gradually approximating the true Q-value.

[0132] The input layer of the Critic network consists of the global state space and the global action space. In this invention, the size of the Critic network input layer is 3×M+4×S. The output layer size is 1, which is the network's evaluation value of the Actor network, also known as the Q-value.

[0133] In the Actor network, the SW selects a scheduling scheme, including a waiting time, based on the state of the data frames. This scheme includes setting the waiting time for the data stream based on the current network state to minimize end-to-end latency and jitter. The target network of the Actor and Critic networks is synchronized using a soft update method, ensuring training stability and improving performance.

[0134] In summary, this invention comprehensively considers the utilization of wired and wireless channels and end-to-end jitter performance in TSN-5G heterogeneous networks, achieving low-latency and low-jitter data transmission, which is particularly suitable for scenarios with high requirements for transmission efficiency and determinism, such as train communication.

[0135] This invention constructs a cooperative scheduling model using a multi-agent reinforcement learning method, effectively addressing the scheduling complexity among multiple base stations and switches. Furthermore, it introduces a graph neural network to model the network topology, further optimizing the performance of the scheduling algorithm under different network topologies. These improvements significantly enhance the efficiency, flexibility, and robustness of network scheduling.

[0136] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0137] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0138] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0139] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. An asynchronous scheduling method for TSN-5G train communication network based on multi-agent reinforcement learning, characterized in that, include: Obtain the topology of the train TSN-5G heterogeneous scenario; Based on the topology of the TSN-5G heterogeneous scenario of the train, a Markov model is constructed using a graph neural network, and the state space, action space and reward function of the Markov model are defined. The data stream information to be processed in the TSN-5G heterogeneous network is input into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm. The train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm treats each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculation, and outputs the scheduling scheme of the data stream to be processed. The acquisition of the topology of the train TSN-5G heterogeneous scenario includes: The train TSN-5G heterogeneous scenario includes onboard 5G terminal equipment (UE), 5G base station (BS), user function plane (UPF), TSN switch (SW), and driver control console (Server). A centralized network configuration with a CNC server for overall control is used. There are N UEs in total, denoted as UE = {ue1, ue2, ..., ue...}. N There are M elements in total, denoted as B = {b1, b2, ..., b}. M There are S elements SW in total, denoted as SW = {sw1, sw2, ..., sw}. S The entire device comprises L data streams, and device U... i It will send a data stream L i Item, denoted as set up The number of data frames sent in the current data stream. The size of a single data frame in the data stream. For data stream sending period, This represents the average latency of the current data stream. The BS status settings include the data frame wireless transmission time T. w Number of BS queued data frames Q i and channel usage U channel The status settings for SW include the data frame arrival time T. a Data frame compliance time T r Accumulated waiting time T for data frames wait Link utilization U of each port of the switch link ; The process of inputting the data stream information to be processed in the TSN-5G heterogeneous network into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm, wherein the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm treats each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs a scheduling scheme for the data stream to be processed, includes: In the MADGNN algorithm of TSN-5G collaborative scheduling, BS and SW are used as agents. Each BS and SW has an independent Actor network and Critic network. The Actor network and Critic network are deployed on the CNC server. The Base Station (BS) obtains the current system's environmental state space through the Critic network. This environmental state space includes channel utilization, data frame queuing count, and data frame wireless transmission time. The BS inputs this environmental state space into its Actor network, which generates actions including data stream transmission frequency and latency. The SW obtains the current device's environmental state space through the Critic network. This environmental state space includes data frame arrival time, compliance time, cumulative latency, and SW link utilization. The SW inputs this environmental state space into its Actor network, which generates actions including data stream latency. After BS and SW perform actions, the system provides a reward value. The system stores the current state, action, reward, and next state as a quadruple in the experience replay pool. When the number of samples in the replay pool reaches a threshold, the CNC server extracts samples from the replay pool and uses the quadruple data in the samples to train and update the Actor network and Critic network.

2. The method according to claim 1, characterized in that, The topology structure based on the heterogeneous scenario of the train TSN-5G utilizes a graph neural network to construct a Markov model, defining the state space, action space, and reward function of the Markov model, including: The Markov decision process is extended to the multi-agent environment of the heterogeneous scheduling scenario of train TSN-5G. The collaborative scheduling problem between SW and BS in TSN-5G is modeled as a Markov game model. In the Markov game model, each base station and switch is treated as an independent agent. Each agent calculates the reward function based on the perceived environmental state and selects different actions. The six-tuple in a Markov game model is defined as follows (M+S, State, Action). M+S ,P,γ,Reward M+S In the system, M+S represents the number of agents, State represents the system state (either the joint state of the agents or a partially observable state of a single agent), Action represents the joint action space of the agents, P represents the state transition function (i.e., the probability distribution of the state value at the next moment, given the current system state and the joint action space Action), and R represents the state transition function. i ,i∈{1,…,M+S} is the reward function of the i-th agent, and γ represents the discount factor; In the train TSN-5G multi-agent data frame scheduling environment, considering multiple switchers and base operators, the state space, action space, and reward function are defined as follows: State Space: In the CNC server, each agent obtains overall load information and some of its own parameters. For agent B in the BS architecture... i The definition of the observed state at time t is shown in equation (3); Representing the BS intelligent agent B i The state is observed at time t. Representative in BS intelligent agent B i The wireless transmission time of the data frame arriving at time t. Representing the BS intelligent agent B i The length of the data frame queue at time t. Representing the BS intelligent agent B i Wireless channel utilization at time t; For SW agent sw i The definition of the observed state at time t is shown in equation (4); Representing the SW intelligent agent sw i The state is observed at time t. Representing the SW intelligent agent sw i The ATS parameter of the data frame arriving at time t—the arrival time. Representing the SW intelligent agent sw i The ATS parameter of the data frame arriving at time t—compliance time. Representing the SW intelligent agent sw i The cumulative waiting time for data frames arriving at time t. Representing the SW intelligent agent sw i Wired link utilization at time t; According to the definition and The global state space is obtained as shown in equation (5); S t This represents the global observation state at time t. The partially observable state space representing the agent. to It is a portion of the observable state space of each BS. to It is a partial observable state space of each SW; Action Space: The action spaces of BS and SW are different. Each BS has the same actions, and each SW has the same actions. BS agent B... i With SW agent sw i The actions that can be selected at time t are: For B i ,action For wireless transmission frequency and latency, for SW i ,action Waiting time; Reward function: Each agent uses a target function as its reward. The transmission delay of a single data frame is shown in equation (1), where T ij Representative device ue i Data stream sent Queuing time in BS For the BS to SW link transmission time, For the link transmission time between SW, The processing time of the data frame in the switch. The time it takes for a data frame to be sent in the switch. The queuing time for data frames in the switch; The objective function is set as shown in equation (2) with the goal of minimizing data frame transmission delay and minimizing data frame transmission jitter:

3. The method according to claim 1, characterized in that, The CNC server extracts samples from the playback pool and uses the quadruple data in the samples to train and update the Actor network and Critic network, including: The CNC server extracts samples from the playback pool and transmits the quadruple data in the samples to the Actor network and the Critic network. The Actor network and the Critic network adopt a dual network structure, using both the current network and the target network. The Critic network uses the quadruples data in the experience replay pool to calculate the evaluation value Q of the Actor network and performs gradient updates based on the TD error, as shown in Equation (6). The MSE loss function calculated using the TD time-series difference error is shown in Equation (7). The parameters are updated using gradient descent. In the formula, The parameters of the Critic network, and their expected values. The data consists of a quadruple sampled from the experience replay pool, including the current state s, the action taken a, and the immediate reward r after execution. i The next state s′ after the transition and the flag d indicating whether it is a terminating state, and the current Q-value function. Estimate the value of taking action a in state s; The Actor network uses the quadruples data from the experience replay pool to calculate the MSE loss function of the TD error, as shown in Equation (8), to calculate the loss function with respect to θ. i The gradient is used to update the parameters using gradient descent, as shown in equation (9); θ i The parameters of the Actor network, and the expected symbol. The expected value of a state drawn from the experience replay pool D is used to estimate the average effect across the entire state space, and the policy gradient. The loss function is expressed as a function of the policy parameter θ. i The derivative of the gradient is used to indicate the direction of parameter optimization; it is included in the gradient formula. The term represents the rate of change of the Q-value function relative to the policy parameters. This indicates that the Actor network is in a given state s. i The output action below; The target network is updated using a soft update method, as shown in equation (10): This represents the parameters of the main Q network. Here, θ represents the parameters of the target Q-network, θ represents the parameters of the master policy network used to select actions, and θ′ represents the parameters of the target policy network. The parameter τ is an update coefficient, between 0 and 1, controlling the influence of the master network parameters on the target network parameters. In the update rule... Parameters of the main Q network and its original value The values ​​of τ and (1-τ) are mixed proportionally, and θ′ is obtained by mixing the parameters θ and their original values ​​θ′ of the main policy network proportionally.

Images