TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning

Abstract

The invention provides a TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning. The method comprises the following steps: constructing a Markov model by utilizing a graph neural network based on a topological structure of a train TSN-5G heterogeneous scene, defining a state space, an action space and a reward function of the Markov model, and inputting to-be-processed data flow information in a TSN-5G heterogeneous network into a train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm, according to the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm, each base station and a switch serve as independent agents, a state space and an action space in a Markov model are updated through iterative operation, and a scheduling scheme of a data stream to be processed is output. The invention provides a comprehensive scheme, the utilization rate of wired and wireless channels in TSN-5G and the end-to-end jitter performance are combined, the purpose of optimizing data transmission is achieved, and optimal control over end-to-end time delay and jitter of a whole vehicle network is achieved.

Description

Technical Field

[0001] The present invention relates to the field of data stream transmission technology, and in particular to a TSN (Time-Sensitive Networking)-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning. Background Art

[0002] The switch identifies and classifies frames based on information such as identifiers at the inbound port. Flows of different priorities are obtained: TT flow (Time-Triggerd), AVB flow (audio and video flow) and BE (Best Effort). The flow rate is limited through flow shaping. Flow scheduling and preemption determine the scheduling order of the flow. Finally, the switch forwards the traffic to the target device or the next switch at the outbound port.

[0003] The purpose of shaping is to reduce network congestion and packet loss rate. The implementation means is to limit the rate of burst traffic and cache it, and control the flow to be sent at a relatively uniform rate. Currently, common shaping technologies are credit-based shaping (CBS) in clock synchronization scenarios and ATS (Asynchronous Traffic Shaping) in asynchronous scenarios.

[0004] Flow scheduling ensures the orderliness of flow forwarding and reduces network congestion. The main indicators to be considered are latency and jitter.

[0005] (1) Time-Aware Shaper (TAS)

[0006] IEEE 802.1Qbv defines TAS and gated list technologies. TAS allocates different transmission time slots to each transmission window to reduce interference between different data streams. The gated structure is used to control the transmission of data frames. The gated table records the opening and closing of all queue gates at a certain moment.

[0007] (2) Cyclic Queuing and Forwarding (CQF)

[0008] IEEE 802.1Qch introduces the CQF mechanism. It allows switches to synchronize frames in a cyclic manner. The following is a brief introduction to the CQF process: Assume that there are two transmission queues and a cyclic period. In the even-numbered period, queue 1 accumulates frames from the switch receiving port, and queue 2 transmits frames accumulated in the odd-numbered period. In the odd-numbered period, queue 1 sends frames accumulated in the even-numbered period, and queue 2 receives frames from the switch receiving port. As long as the cyclic period is set reasonably, it is guaranteed that any data stream that is sensitive to latency can be scheduled and transmitted within the cyclic period.

[0009] At present, the technical solutions for flow scheduling in the prior art include: scheduling algorithms in synchronous and asynchronous scenarios according to whether the clocks are synchronized.

[0010] Scheduling Algorithm in Clock Synchronization Scenario

[0011] Research on scheduling algorithms in synchronous scenarios includes TSN and TSN-5G fusion scenarios.

[0012] (1) Switch queue length prediction

[0013] The enhanced Long Short-Term Memory (LSTM) network is used to predict the length of the queue in each switch. LSTM is a variant of the Recurrent Neural Network (RNN) that can effectively process temporal and spatial sequence data. Compared with RNN, LSTM can selectively store information through gated transposition.

[0014] In this solution, the model uses network topology, traffic type, and queue length as input, and queue length as output to achieve a prediction task. That is, the number of queued frames in the future is predicted based on the current network status and the number of queued frames on the switch. This is used as an indicator of network congestion. The routing strategy is then adjusted based on the predicted results to achieve dynamic routing. However, using queue length as a representative of network congestion is too one-sided and cannot maximize the reduction of end-to-end delay and jitter.

[0015] (2) TSN-5G Particle Swarm Reinforcement Learning Algorithm

[0016] An integrated industrial TSN-5G network architecture is proposed. Double Q-Learning Hierarchical Particular Swarm Optimization (DQHPSO) is proposed. The particle swarm optimization algorithm simulates the search behavior of a group of particles in a multidimensional space. Each particle represents a potential solution and updates its position and velocity through the historical best position of individual particles and the group.

[0017] Taking the scheduling success rate as the optimization target, each switch can successfully transmit data within the transmission window, that is, it meets the CQF cycle period, thus ensuring the final delay and jitter indicators.

[0018] (3) TSN-5G joint scheduling takes 5G channels into consideration

[0019] This method considers the 5G time slot utilization and SINR signal-to-noise ratio (SINR), that is, the impact of signal quality on data transmission, and uses a reinforcement learning algorithm to improve signal utilization. 5G uses millisecond time slots, and this method introduces micro-time slots to improve channel utilization.

[0020] The shortcomings of the above-mentioned flow scheduling technical solutions in the prior art include: In the existing TSN-5G synchronous or asynchronous network joint scheduling methods, some studies focus on the impact of jitter, while others focus on improving channel utilization to optimize delay performance. However, considering jitter alone may lead to a significant increase in latency, while focusing only on channel utilization may cause larger data frame transmission jitter. In actual train communication scenarios, data transmission needs to meet the dual requirements of low latency and low jitter to ensure the efficiency and determinism of communication.

[0021] Existing reinforcement learning methods for TSN-5G heterogeneous scenarios are mostly based on single-agent architectures. However, in such scenarios, each base station and each switch, as independent scheduling units, face different scheduling tasks. At the same time, there are coordination and constraint relationships between base stations, base stations and switches, and switches and switches. Therefore, the present invention designs a scheduling method based on multi-agent reinforcement learning, which is suitable for TSN-5G heterogeneous network scenarios, and aims to build a more comprehensive and collaborative scheduling model to improve the overall performance of the system.

[0022] In the existing TSN-5G heterogeneous scenarios, reinforcement learning methods often only accelerate the convergence process of optimization algorithms through reinforcement learning from the perspective of network topology relevance, without fully considering the potential impact of network topology on scheduling results. Due to differences in network topology structures, network traffic load, scheduling decisions, and bandwidth resource occupancy will be directly affected. To address this problem, the present invention introduces a graph neural network, which aims to build a more accurate network topology model, thereby improving the performance of scheduling methods under different topologies. Summary of the invention

[0023] An embodiment of the present invention provides a TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning to achieve optimal control of the end-to-end delay and jitter of the entire vehicle network.

[0024] In order to achieve the above object, the present invention adopts the following technical scheme.

[0025] A TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning, comprising:

[0026] Obtain the topology of the train TSN-5G heterogeneous scenario;

[0027] Based on the topological structure of the train TSN-5G heterogeneous scenario, a Markov model is constructed using a graph neural network, and a state space, action space, and reward function of the Markov model are defined;

[0028] The data stream information to be processed in the TSN-5G heterogeneous network is input into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm. The train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm takes each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs a scheduling plan for the data stream to be processed.

[0029] Preferably, the obtaining of the topological structure of the train TSN-5G heterogeneous scenario includes:

[0030] The train TSN-5G heterogeneous scenario includes on-board 5G terminal equipment UE, 5G base station BS, user function plane UPF, TSN switch SW and driver control console Server. A centralized network configuration CNC server is used for overall control. There are N UEs in total, denoted as UE = {ue1, ue2, ..., ue N}, there are M BSs in total, denoted as B = {b1, b2, …, b M}; there are S SW in total, recorded as SW = {sw1, sw2, ..., sw S}, all devices include L data streams, device U i Will send data stream L i , recorded as set up is the number of data frames sent by the current data stream, The size of a single data frame of the data stream. is the data stream sending period, is the average delay of the current data flow;

[0031] Set the BS status content including the data frame wireless transmission time T w , the number of BS queued data frames Q i and channel usage U channel , set the SW status content including the data frame arrival time T a , data frame compliance time T r , Data frame cumulative waiting time T wait and the link utilization U of each switch port link .

[0032] Preferably, the topological structure based on the train TSN-5G heterogeneous scenario uses a graph neural network to construct a Markov model, and defines the state space, action space and reward function of the Markov model, including:

[0033] The Markov decision process is extended to the multi-agent environment in the train TSN-5G heterogeneous scheduling scenario, and the SW and BS collaborative scheduling problem in TSN-5G is modeled as a Markov game model. In the Markov game model, each base station and switch is regarded as an independent agent. Each agent calculates the reward function according to the perceived environmental state and selects different actions.

[0034] The six-tuple in the Markov game model is defined as follows (M+S, State, Action M+S ,P,γ,Reward M+S ), where M+S is the number of agents in the system, State is the system state, which refers to the joint state of multiple agents or the partially observable state of a certain agent, Action is the joint action space of the agents; P is the state transfer function, that is, the probability distribution of the state value at the next moment given the current system state State and the multi-agent joint action space Action; R i ,i∈{1,…,M+S} is the reward function of the i-th agent;

[0035] In the train TSN-5G multi-agent data frame scheduling environment, the state space, action space, and reward function are defined by considering multiple SWs and BSs as follows:

[0036] State space: In the CNC server, each agent obtains the overall load information and its own partial parameters. i The definition of the observed state at time t is shown in formula (3);

[0037]

[0038] Represents BS agent B i Observe the state at time t, Represents BS agent B i The wireless transmission time of the data frame arriving at time t, Represents BS agent B i The length of the data frame queue at time t (due to network performance limitations, data frames will be queued in the BS agent), Represents BS agent B i Wireless channel utilization at time t;

[0039] For SW agent sw i The definition of the observed state at time t is shown in formula (4);

[0040]

[0041] Represents SW agent sw i Observe the state at time t, Represents the SW agent sw i ATS parameters of the data frame arriving at time t - arrival time, Represents SW agent sw i ATS parameters of the data frame arriving at time t - compliance time, Represents SW agent sw i The accumulated waiting time of the data frame arriving at time t, Represents SW agent sw i Wired link utilization at time t;

[0042] According to the definition and The global state space is obtained as shown in formula (5);

[0043]

[0044] S t represents the global observation state at time t, represents part of the agent's observable state space, to is the partially observable state space of each BS, to is the partially observable state space of each SW;

[0045] Action space: The action space of BS and SW is different. The actions of each BS are the same, and the actions of each SW are the same. BS agent B i With SW agent sw i The actions that can be chosen at time t are For B i ,action is the wireless transmission frequency and waiting time, for sw i ,action For waiting time;

[0046] Reward function: Each agent uses the objective function as a reward. The transmission delay of a single data frame is shown in formula (1), where T ij Representative equipment ue i Outgoing data stream is the queuing time in BS, is the transmission time of the BS to SW link, is the link transmission time between SWs, The processing time of the data frame in SW, The time when the data frame is sent in SW, The time for data frames to queue in SW;

[0047]

[0048] The objective function is set to minimize the data frame transmission delay and minimize the data frame transmission jitter as shown in formula (2):

[0049]

[0050] Preferably, the data stream information to be processed in the TSN-5G heterogeneous network is input into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm, and the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm takes each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs the scheduling scheme of the data stream to be processed, including:

[0051] In the MADGNN algorithm of TSN-5G collaborative scheduling, BS and SW are used as intelligent agents. Each BS and SW has an independent actor network and critic network, which are deployed in the CNC server.

[0052] BS obtains the current system's environmental state space through the Critic network. The environmental state space of the current system includes channel utilization, the number of data frame queues, and the wireless transmission time of data frames. The environmental state space of the current system is input into the Actor network of the BS. The BS generates an action through the Actor network. The action includes the transmission frequency and waiting time of the data stream. SW obtains the current device's environmental state space through the Critic network. The environmental state space of the current device includes the arrival time, compliance time, accumulated waiting time, and SW link utilization of the data frame. The environmental state space of the current device is input into the Actor network of the SW. The SW generates an action through the Actor network. The action includes the waiting time of the data stream.

[0053] After BS and SW perform actions, the system feeds back reward values. The system stores the current state, action, reward, and next state into a four-tuple and stores it in the experience replay pool. When the number of samples in the replay pool reaches the threshold, the CNC server extracts samples from the replay pool and uses the four-tuple data in the samples to train and update the Actor network and Critic network.

[0054] Preferably, the CNC server extracts samples from the playback pool, and uses the quadruple data in the samples to train and update the Actor network and the Critic network, including:

[0055] The CNC server extracts samples from the playback pool and transmits the four-tuple data in the samples to the Actor network and the Critic network. The Actor network and the Critic network adopt a dual network structure, using two forms: the current network and the target network.

[0056] The Critic network uses the four-tuple data in the experience replay pool to calculate the evaluation value Q of the Actor network, and performs a gradient update based on the TD error, as shown in formula (6). The MSE loss function calculated by the TD temporal difference error is shown in formula (7). The parameters are updated using gradient descent

[0057]

[0058] In the above formula, Represents the parameters of the Critic network, expected value is the four-tuple data sampled from the experience replay pool, including the current state s, the action a taken, the immediate reward r after execution, the next state s′ after transfer, and the mark d indicating whether it is the terminal state. The current Q value function Estimate the value of taking action a in state s;

[0059] The Actor network uses the quadruple data in the experience replay pool to calculate the MSE loss function of the TD error, as shown in formula (8), and calculates the MSE loss function about θ i The gradient of , using gradient descent to update the parameters, as shown in formula (9);

[0060]

[0061] θ i Represents the parameters of the Actor network, expected symbol Represents the expectation of extracting states from the experience replay pool D, which is used to estimate the average effect of the entire state space, the policy gradient Denotes the loss function for the policy parameter θ i The derivative of is used to indicate the optimization direction of the parameters. The gradient formula contains The term represents the rate of change of the Q-value function relative to the policy parameters, Represents the Actor network in a given state s i The output action below;

[0062] The target network is updated by soft update, as shown in formula (10):

[0063]

[0064] represents the parameters of the main Q network, represents the parameters of the target Q network, θ represents the parameters of the main policy network, which is used to select actions, and θ′ is the parameter of the target policy network. The parameter τ is the update coefficient, which is between 0 and 1 and controls the influence of the main network parameters on the target network parameters. In the update rule, The parameters of the main Q network and its original value It is obtained by mixing in proportion τ and (1-τ), and θ′ is obtained by mixing in proportion the parameter θ of the main strategy network and its original value θ′.

[0065] It can be seen from the technical solutions provided by the above-mentioned embodiments of the present invention that the present invention proposes a comprehensive solution that combines the utilization rate of wired and wireless channels in TSN-5G with the end-to-end jitter performance to achieve the purpose of optimizing data transmission.

[0066] Additional aspects and advantages of the present invention will be given in part in the following description, which will become obvious from the following description, or may be learned through practice of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0068] Figure 1 A schematic diagram of a train TSN-5G heterogeneous scenario provided by an embodiment of the present invention;

[0069] Figure 2 A processing flow chart of a TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning provided by an embodiment of the present invention;

[0070] Figure 3 A data frame transmission flow chart provided by an embodiment of the present invention;

[0071] Figure 4 A schematic diagram of the implementation principle of a TSN-5G scheduling algorithm based on MADGNN (Multi-Agent Deep deterministic policy gradient with Graph Neural Network) in a Markov game model provided in an embodiment of the present invention. DETAILED DESCRIPTION

[0072] The embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals throughout represent the same or similar elements or elements having the same or similar functions. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and cannot be interpreted as limiting the present invention.

[0073] It will be understood by those skilled in the art that, unless expressly stated, the singular forms "one", "said", and "the" used herein may also include plural forms. It should be further understood that the term "comprising" used in the specification of the present invention refers to the presence of the features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or groups thereof. It should be understood that when we refer to an element as being "connected" or "coupled" to another element, it may be directly connected or coupled to the other element, or there may be intermediate elements. In addition, the "connection" or "coupling" used herein may include wireless connection or coupling. The term "and / or" used herein includes any unit and all combinations of one or more associated listed items.

[0074] It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as those generally understood by those skilled in the art in the art to which the present invention belongs. It should also be understood that terms such as those defined in common dictionaries should be understood to have meanings consistent with the meanings in the context of the prior art, and will not be interpreted with idealized or overly formal meanings unless defined as herein.

[0075] To facilitate understanding of the embodiments of the present invention, several specific embodiments will be further explained below with reference to the accompanying drawings, and each embodiment does not constitute a limitation on the embodiments of the present invention.

[0076] The present invention uses 5G as a transparent bridge solution, which is only used for data transmission. The present invention connects the TSN network with the 5G network, and the network inside the entire system is relatively independent. The determinism of service transmission is achieved by segmentation to improve the end-to-end service transmission quality. The relevant marks of the TSN network for traffic configuration are retained, and the 5G encapsulation is stripped after remote transmission through the 5G network system. After entering the collaborative service system, deterministic transmission is still performed according to the TSN traffic scheduling type. The business system networks at both ends are TSN networks, and the 5G air interface, 5G bearer network, and 5G core network in the middle are 5G networks.

[0077] In the train onboard network communication scenario, the introduction of 5G wireless technology to reduce wiring can enhance the flexibility of the system. However, wireless technology faces the problem of uncertainty in end-to-end data transmission due to factors such as complex environment and complicated equipment, so TSN technology is used to enhance determinism. Existing synchronization methods have problems such as clock asynchrony between TSN and 5G heterogeneous networks and additional overhead. Existing asynchronous methods have the problem of too long waiting time causing too high delay, without considering the constraint relationship between different switches, and without considering the problem that the switch topology will affect the scheduling results.

[0078] The present invention proposes a method for joint asynchronous scheduling of base stations and switches that takes into account the collaborative constraints between base stations, base stations and switches, and switches, as well as the influence of the overall network topology, to reduce the delay and jitter indicators of data transmission. It alleviates the uncertainty problem of data transmission caused by the introduction of wireless transmission in industrial Ethernet. Here, base stations and switches are regarded as two types of agents in a multi-agent environment.

[0079] An embodiment of the present invention provides a train TSN-5G heterogeneous scenario such as Figure 1 As shown. The equipment in the whole system includes vehicle-mounted 5G terminal equipment (User Equipment, UE), 5G base station (Base Station, BS), UPF (User Function Plane, located in the 5G core network), TSN switch (TSN Switch, SW), and driver console (Server). There are N vehicle-mounted 5G terminal equipment, denoted as UE = {ue1, ue2, ..., ue N},exist Figure 1 In the example, the traction control unit (TCU), brake control unit (BCU) and door control unit (DCU) are taken as examples; there are M BSs in total, denoted as B = {b1, b2, ..., b M}; there are S SW in total, recorded as SW = {sw1, sw2, ..., sw S}. All devices include L data streams, device U i Will send data stream L i , recorded as In addition to the devices described above, CNC (Centralized Network Configuration) devices will also be used for overall control to reduce the computational burden of training and deploying models for switches and base stations. is the number of data frames sent by the current data stream, The size of a single data frame of the data stream. is the data stream sending period, The average delay of the current data flow.

[0080] For BS, the status content can be defined such as the wireless transmission time of the data frame T w , which indicates the transmission time of data frames under the condition that they may be interfered by the quality of wireless channel. This is the main reason for the transmission jitter of the whole transmission model. The number of BS queued data frames Q i , denoted as B i The number of data frames currently queued; channel usage U channel , indicating the current channel occupancy, so as to facilitate further improvement of channel utilization through algorithms.

[0081] For SW, the status content can be defined such as the data frame arrival time T a , because it is a heterogeneous scenario, the arrival time here is not the global synchronization time, and will work together with the compliance time; the data frame compliance time T r The compliance time is a parameter calculated by the ATS algorithm and is related to the arrival time T a The combined representation of the current queueing system's queueing status facilitates the adjustment of the current SW load through model actions; the accumulated waiting time T of the data frame wait , this parameter represents the waiting time of the data frame in the BS and the SW that has passed through, indicating the waiting time of the data frame through the joint scheduling model, so as to better predict and reason about the actions that the SW will perform; the link utilization rate of each port of the switch U link , represents the utilization of the SW egress port, thus indicating the link utilization of different traffic paths, so as to facilitate the scheduling model in the SW to improve the link utilization and balance the link utilization between different SWs, to prevent some devices from being overloaded while the utilization of other devices is too low.

[0082] The processing flow chart of a TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning provided by an embodiment of the present invention is as follows: Figure 2 As shown, the process includes the following steps:

[0083] Step S10: Obtain the topological structure of the train TSN-5G heterogeneous scenario.

[0084] Step S20: constructing a Markov model using a graph neural network based on the topological structure of the train TSN-5G heterogeneous scenario, and defining the state space, action space and reward function of the Markov model;

[0085] Step S30: Input the data stream information to be processed in the above-mentioned TSN-5G heterogeneous network into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling (MADGNN) algorithm. The MADGNN algorithm regards each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative operations, and outputs the scheduling plan for the data stream to be processed.

[0086] The data flow scheduling scheme is the action generated in the Markov model of the reinforcement learning algorithm. The base station generates actions - transmission frequency and waiting time - by obtaining the states of channel utilization, number of data frames queued, and wireless transmission time of data frames. For transmission frequency, the resources available to 5G base stations are time and frequency resources. This algorithm uses base stations to allocate frequency resources for data frame transmission, and different frequency allocation schemes will be generated according to the channel utilization and number of data frames queued at the base station. For waiting time, also called holding time, the base station generates waiting time based on the aforementioned states, and the data frame will be held at the base station for a period of time.

[0087] Similarly, the waiting time is set in the switch SW device, and the switch obtains the data frame arrival time, data frame compliance time, data frame cumulative waiting time, and SW link utilization to generate the waiting time. The waiting time is designed in both BS and SW to ensure load balancing.

[0088] A data frame transmission process provided by an embodiment of the present invention is as follows: Figure 3 As shown. The data frame is generated by the UE, encapsulated into a 5G frame by the DSTT (Device-side TSN Translator) component, and then wirelessly transmitted to the 5G base station. The 5G base station executes the scheduling algorithm to improve link utilization. The NWTT (Network-side TSN Translator) component of the core network decapsulates the 5G frame. The data frame is transmitted to the TSN network and enters the SW. The SW executes the TSN-5G scheduling algorithm based on MADGNN, allowing the corresponding data frame to wait for a period of time to alleviate the jitter problem caused by wireless transmission. Finally, it will reach the target device after passing through multiple switches. The transmission of this data frame is completed. BS and SW respectively obtain different state spaces such as equations (3) and (4) to make decisions. Decisions are all generated actions, and the scheduling algorithm is still executed.

[0089] According to the above single data frame transmission process, the transmission delay of a single data frame can be obtained as shown in formula (1), where T ij Representative equipment ue i Outgoing data stream is the queuing time in BS, is the transmission time of the BS to SW link, is the link transmission time between SWs, The processing time of the data frame in SW, The time when the data frame is sent in SW, The time that a data frame is queued in SW.

[0090]

[0091] BS and SW will generate actions based on the system status. BS will generate wireless transmission frequency and waiting time based on channel utilization, number of data frame queues and wireless transmission time of data frames. SW will generate waiting time based on data frame arrival time, data frame compliance time, data frame cumulative waiting time and SW link utilization.

[0092] The present invention aims to minimize data frame transmission delay and data frame transmission jitter to improve network transmission performance and solve the multi-objective optimization problem of delay and jitter in the train TSN-5G heterogeneous network. The above objective function is defined as shown in formula (2).

[0093]

[0094] Among them, λ and μ are hyperparameters, which are dynamically adjusted according to different training results (different delay and jitter indicators). ij Representative equipment ue i Outgoing data stream The sum of the time taken to reach the destination device is the delay of the data flow. Representative i Device data flow The average delay of The jitter representing all devices is averaged.

[0095] A Markov game model for a train TSN-5G communication scenario provided by an embodiment of the present invention includes: in the train TSN-5G heterogeneous scheduling scenario, each SW and BS is used as an agent, the Markov decision process is extended to a multi-agent environment, and the collaborative scheduling problem of SW and BS in TSN-5G is modeled as a Markov game model. In the Markov game model, multiple agents (that is, SW and BS) will interact with each other, and there are mutual constraints and collaborations between BSs, between BS and SW, and between SWs, and there is uncertainty in the system. Each agent will choose different actions according to the perceptible environmental state and affect the next state generation. Due to the uncertainty of the system, the reward function that each agent needs to calculate depends not only on the reward generated by its own action, but also on the reward function obtained by comprehensively considering the actions of other agents, with the goal of maximizing long-term rewards. The entire system will balance the distribution of reward values ​​for each agent to maximize the benefits of the system.

[0096] In the Markov game model, a six-tuple is usually used for formal description. Substituting into the scenario defined in the present invention, the six-tuple is described as follows (M+S, State, Action M+S ,P,γ,Reward M+S ), where M+S is the number of agents in the system, which is the number of BSs and SWs in this scenario; State is the system state, which generally refers to the joint state of multiple agents or the partially observable state of a certain agent. In this scenario, due to the use of CNC servers, the joint state will be used as the overall input; Action is the joint action space of the agents; P is the state transfer function, that is, the probability distribution of the state value at the next moment given the current system state State and the multi-agent joint action space Action; R i ,i∈{1,…,M+S} is the reward function of the i-th agent. Since the system goal will comprehensively consider the delay and jitter of the data frame arrival, the reward function of each agent in the present invention is consistent.

[0097] In the train TSN-5G multi-agent data frame scheduling environment, the state space, action space, and reward function are defined by considering multiple SWs and BSs as follows:

[0098] State space: In the CNC server, each agent can obtain the overall load information and some of its own parameters. i The definition of the observed state at time t is as shown in formula (3).

[0099]

[0100] Represents BS agent B iObserve the state at time t, Represents BS agent B i The wireless transmission time of the data frame arriving at time t, Represents BS agent B i The length of the data frame queue at time t (due to network performance limitations, data frames will be queued in the BS agent), Represents BS agent B i Wireless channel utilization at time t.

[0101] For SW agent sw i The definition of the observed state at time t is as shown in equation (4).

[0102]

[0103] Represents SW agent sw i Observe the state at time t, Represents the SW agent sw i ATS parameters of the data frame arriving at time t - arrival time, Represents SW agent sw i ATS parameters of the data frame arriving at time t - compliance time, Represents SW agent sw i The accumulated waiting time of the data frame arriving at time t, Represents SW agent sw i Wired link utilization at time t.

[0104] According to the definition and The global state space can be obtained as shown in formula (5).

[0105]

[0106] S t represents the global observation state at time t, represents part of the agent's observable state space, to is the partially observable state space of each BS, to It is the partially observable state space of each SW.

[0107] Action space: The action space of BS and SW is different. The actions of each BS and each SW are the same. i With SW agent sw i The actions that can be chosen at time t are For B i,action is the wireless transmission frequency and waiting time. i ,action For waiting time.

[0108] Reward function: Each agent uses an objective function as a reward.

[0109] In order to reduce the end-to-end delay and jitter indicators of TSN-5G heterogeneous network data frame transmission, global state information, action space and reward function are defined in the Markov game model. The implementation principle of the TSN-5G scheduling algorithm based on MADGNN in a Markov game model provided by an embodiment of the present invention is as follows: Figure 4 As shown in the figure. Each BS and SW is an intelligent agent with an independent Actor network and Critic network. Since the system uses a CNC server. The Actor network and the Critic network are both deployed in the CNC server, and the network training and deployment processes are centrally executed in the CNC server. For multi-agent algorithms, there are centralized training and decentralized execution. That is, the global information is obtained through the Critic network to conduct collaborative centralized training with the Actor. After the training is completed, each intelligent agent makes waiting time decisions or frequency decisions through an independent Actor network.

[0110] The Actor network uses a layered graph neural network, and the Critic network uses a holistic graph neural network. Agent B i or sw i The Actor strategy network is denoted as That is, the Actor network can be based on the state of the agent itself g i Make a decision i The Critic network calculates the Q value and performs training updates on the Actor network. The algorithm learns both the value function and the policy function during the training process. The Critic is the value network of the intelligent agent, denoted as

[0111] Both the Actor network and the Critic network use dual networks, that is, using the current network and the target network, to avoid the overestimation problem caused by using the same network when updating the Q value. i or sw i The target Actor network is denoted as Used to get the next action through strategy The target critic network is denoted as Used to calculate the target Q value.

[0112] The training data is obtained through the interaction between the agent and the environment. For example, in the data frame transmission process, the UE sends data, and at time t the BS or SW will receive the data frame and make waiting time decisions or frequency decisions based on the observed state. The global state S and action set A are formed. The executed actions will be fed back to the environment to obtain the reward set R for each agent, and the environment state will go from S at time t to S' at time t+1. In this way, a four-tuple (S, A, R, S') is generated. This data will be put into the experience revisit pool, and after accumulating a certain amount, the Critic network and the Actor network will be trained.

[0113] For the Critic network, the MSE loss function of the two is constructed using the TD error (Temporal Difference Error) as shown in formula (6), and the parameters are updated using gradient descent As shown in formula (7).

[0114]

[0115] In the above formula, represents the parameters of the Q-value network (i.e., the Critic network), which estimates the value of a state-action pair by optimizing these parameters. The formula contains the expected value of a set of data sampled from the experience replay pool It includes the current state s, the action a taken, the immediate reward r after execution, the next state s' after the transfer, and the mark d whether it is the terminal state. Current Q value function Estimate the value of taking action a in state s. The target Q value is partly determined by the immediate reward r i , discount factor γ and optimal Q-value estimate for the next state The discount factor is used to control the importance of future rewards. If the next state is a terminal state, future rewards will not be taken into account. The goal of the loss function is to minimize the mean square error between the current Q value and the target Q value, so that the network's estimate gradually approaches the true value. It represents the changing trend of the loss function with respect to the Q-value network parameters, through which the parameters can be adjusted to minimize the loss.

[0116] For the Actor network, when calculating the forward propagation of each agent's own Actor, each agent only uses the local observation state as the input of the Actor network to obtain deterministic actions. The MSE loss function for calculating the TD error is as shown in Equation (8) and is calculated with respect to θ i The gradient of , and the parameters are updated using gradient descent as shown in formula (9).

[0117]

[0118] θi Represents the parameters of the policy network (i.e., Actor network), which are optimized by minimizing the loss function to improve the performance of the policy. Represents the expectation of extracting states from the experience replay pool D, which is used to estimate the average effect of the entire state space. Same as above, is the Q-value function, which estimates the value of taking action a in state s. Policy Gradient The loss function is expressed as i The derivative of is used to indicate the optimization direction of the parameters. The gradient formula contains The term represents the rate of change of the Q-value function relative to the policy parameters, and there is Indicates that the strategy is in a given state s i These terms work together to calculate the policy gradient, which is used to update the parameters θ of the policy network. i , thereby optimizing the behavior of the agent.

[0119] The target network is updated by soft updating, as shown in formula (10).

[0120]

[0121] represents the parameters of the main Q network, which are frequently updated directly through the training data, and The parameters of the target Q network are updated slowly, mainly to provide a stable target value during training. θ represents the parameters of the main policy network, which is used to select actions, while θ′ is the parameter of the target policy network, which is similar to the target Q network and is used to provide a more stable policy target. The parameter τ is the update coefficient, which is between 0 and 1 and controls the degree of influence of the main network parameters on the target network parameters. In the update rule, The parameters of the main Q network and its original value The target network is obtained by mixing the parameters θ and (1-τ) in proportion. Similarly, θ′ is obtained by mixing the parameters θ of the main policy network and its original value θ′ in proportion. In this way, by only making small parameter updates each time, the target network can more stably approach the performance of the main network.

[0122] The processing process of the TSN-5G heterogeneous collaborative scheduling MADGNN algorithm is shown in Algorithm 1. The algorithm initializes the agent set and the experience replay pool. Through multiple rounds of iterative training, each agent (BS or SW) obtains actions through the Actor network after observing the current environment state in each time step t of each round, such as the frequency allocation of the base station or the data frame waiting time of the switch. After the actions of all agents are executed, the system will feedback the reward value and enter the next environment state. The current state, action, reward, and next state are stored in the experience replay pool. When the experience replay pool reaches a certain number of samples, the network is updated, and the parameters of the Actor and Critic networks are updated based on the samples randomly drawn from the replay pool, and the target network is synchronized through the soft update method until the end of the algorithm.

[0123]

[0124]

[0125] In the MADGNN algorithm of TSN-5G collaborative scheduling, each intelligent agent (including base station BS and switch SW) uses an independent Actor network to make decisions. The base station generates actions - transmission frequency and waiting time based on the current environmental status (such as channel utilization, number of data frame queues, and wireless transmission time of data frames). Among them, the transmission frequency is allocated by the frequency resources of the base station under a given state to adapt to different channel utilization and queuing conditions; the waiting time (or holding time) determines the time the data frame queues at the base station to ensure smooth traffic.

[0126] Similarly, the switch generates corresponding waiting times based on the arrival time, compliance time, accumulated waiting time, and SW link utilization of the data frame through the Actor network. These waiting times are designed to maintain load balance between the BS and SW and alleviate jitter problems that may occur during transmission. The switch will schedule the data flow according to the set waiting time so that the data frame maintains a steady rhythm on the entire transmission link.

[0127] The specific Actor network and Critic network are independent neural networks, such as Figure 4 As shown in the figure. The Critic network uses the global network to extract the global network topology features, and BS, UPF, and SW are used as nodes of the Critic network. The Actor network uses the local graph of the corresponding device to extract the local network topology features, and connects the BS or SW agents and the adjacent connected agents as nodes in the Actor network.

[0128] The input layer of the BS's Actor network is also a state space, which is the channel utilization, the number of data frame queues, and the wireless transmission time of the data frame; the output layer is also an action space, with two actions: wireless transmission frequency and waiting time. The input layer of the SW's Actor network is the arrival time of the data frame, the compliance time, the accumulated waiting time, and the SW link utilization; the output layer has only one action - waiting time, which is the waiting time for the data frame to be kept in the SW.

[0129] After executing the action, the system will feedback the reward value, which is calculated based on indicators such as transmission delay and jitter. That is, the system uses formula (2) to calculate the reward value to reflect the quality of the agent's behavior. The system enters the next environmental state, and the next state can be directly input according to the actual state of the system at the next moment (in this model, the next moment of a certain agent BS or SW is defined as the state when the next data frame arrives at the agent).

[0130] The system stores the current state, action, reward, and next state into a four-tuple in the experience replay pool. When the number of samples in the replay pool reaches a threshold, the algorithm starts to extract samples from it for training the network.

[0131] The Critic network updates the Q value by evaluating the quality of the agent's actions to assist the training of the Actor network. The Critic network calculates the value of the state-action pair and provides a goal for the Actor network to help the agent choose the best action strategy. By using the quadruple data in the experience replay pool, the Critic network performs a gradient update based on the TD error, gradually approaching the true Q value.

[0132] The input layer of the Critic network is the global state space and the global action space. In the present invention, the input layer size of the Critic network is 3×M+4×S. The output layer size is 1, which is the evaluation value of the network to the Actor network, also known as the Q value.

[0133] In the Actor network, SW selects a scheduling scheme such as waiting time according to the state of the data frame. This scheme includes setting the waiting time for the data stream based on the current network state to minimize end-to-end delay and jitter. The target network of the Actor and Critic networks is synchronized through soft updates to ensure the stability of training and performance improvement.

[0134] In summary, the present invention comprehensively considers the utilization of wired and wireless channels and end-to-end jitter performance in the TSN-5G heterogeneous network, and realizes low-latency and low-jitter data transmission, which is particularly suitable for scenarios with high requirements on transmission efficiency and determinism, such as train communications.

[0135] This paper constructs a collaborative scheduling model through a multi-agent reinforcement learning method to effectively deal with the scheduling complexity between multiple base stations and switches, and introduces a graph neural network to model the network topology, further optimizing the performance of the scheduling algorithm under different network topologies. These improvements significantly improve the efficiency, flexibility, and robustness of network scheduling.

[0136] Those skilled in the art can understand that the accompanying drawings are only schematic diagrams of an embodiment, and the modules or processes in the accompanying drawings are not necessarily required to implement the present invention.

[0137] It can be known from the description of the above implementation methods that those skilled in the art can clearly understand that the present invention can be implemented by means of software plus a necessary general hardware platform. Based on such an understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium such as ROM / RAM, a magnetic disk, an optical disk, etc., and includes a number of instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in the various embodiments of the present invention or certain parts of the embodiments.

[0138] Each embodiment in this specification is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device or system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0139] The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by a person skilled in the art within the technical scope disclosed by the present invention should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention should be based on the protection scope of the claims.

Claims

1. A TSN-5G train communication network asynchronous scheduling method based on multi-agent reinforcement learning, characterized in that: include: Obtain the topology of the train TSN-5G heterogeneous scenario; Based on the topological structure of the train TSN-5G heterogeneous scenario, a Markov model is constructed using a graph neural network, and a state space, action space, and reward function of the Markov model are defined; The data stream information to be processed in the TSN-5G heterogeneous network is input into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm. The train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm takes each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs a scheduling plan for the data stream to be processed.

2. The method according to claim 1, characterized in that The method of obtaining the topological structure of the train TSN-5G heterogeneous scenario includes: The train TSN-5G heterogeneous scenario includes on-board 5G terminal equipment UE, 5G base station BS, user function plane UPF, TSN switch SW and driver control console Server. A centralized network configuration CNC server is used for overall control. There are N UEs in total, denoted as UE = {ue1, ue2, ..., ue N }, there are M BSs in total, denoted as B = {b1, b2, …, b M }; there are S SW in total, recorded as SW = {sw1, sw2, ..., sw S }, all devices include L data streams, device U i Will send data stream L i , recorded as set up is the number of data frames sent by the current data stream, The size of a single data frame of the data stream. is the data stream sending period, is the average delay of the current data flow; Set the BS status content including the data frame wireless transmission time T w , the number of BS queued data frames Q i and channel usage U channel , set the SW status content including the data frame arrival time T a , data frame compliance time T r , Data frame cumulative waiting time T wait and the link utilization U of each switch port link .

3. The method according to claim 2, characterized in that The topological structure based on the train TSN-5G heterogeneous scenario uses a graph neural network to construct a Markov model, and defines the state space, action space and reward function of the Markov model, including: The Markov decision process is extended to the multi-agent environment in the train TSN-5G heterogeneous scheduling scenario, and the SW and BS collaborative scheduling problem in TSN-5G is modeled as a Markov game model. In the Markov game model, each base station and switch is regarded as an independent agent. Each agent calculates the reward function according to the perceived environmental state and selects different actions. The six-tuple in the Markov game model is defined as follows (M+S, State, Action M+S ,P,γ,Reward M+S ), where M+S is the number of agents in the system, State is the system state, which refers to the joint state of multiple agents or the partially observable state of a certain agent, Action is the joint action space of the agents; P is the state transfer function, that is, the probability distribution of the state value at the next moment given the current system state State and the multi-agent joint action space Action; R i ,i∈{1,…,M+S} is the reward function of the i-th agent; In the train TSN-5G multi-agent data frame scheduling environment, the state space, action space, and reward function are defined by considering multiple SWs and BSs as follows: State space: In the CNC server, each agent obtains the overall load information and some of its own parameters. i The definition of the observed state at time t is shown in formula (3); Represents BS agent B i Observe the state at time t, Represents BS agent B i The wireless transmission time of the data frame arriving at time t, Represents BS agent B i The length of the data frame queue at time t (due to network performance limitations, data frames will be queued in the BS agent), Represents BS agent B i Wireless channel utilization at time t; For SW agent sw i The definition of the observed state at time t is shown in formula (4); Represents SW agent sw i Observe the state at time t, Represents the SW agent sw i ATS parameters of the data frame arriving at time t - arrival time, Represents SW agent sw i ATS parameters of the data frame arriving at time t - compliance time, Represents SW agent sw i The accumulated waiting time of the data frame arriving at time t, Represents SW agent sw i Wired link utilization at time t; According to the definition and The global state space is obtained as shown in formula (5); S t represents the global observation state at time t, represents part of the agent's observable state space, to is the partially observable state space of each BS, to is the partially observable state space of each SW; Action space: The action space of BS and SW is different. The actions of each BS are the same, and the actions of each SW are the same. BS agent B i With SW agent sw i The actions that can be chosen at time t are For B i ,action is the wireless transmission frequency and waiting time, for sw i ,action For waiting time; Reward function: Each agent uses the objective function as a reward. The transmission delay of a single data frame is shown in formula (1), where T ij Representative equipment ue i The data stream sent i j , is the queuing time in BS, is the transmission time of the BS to SW link, is the link transmission time between SWs, The processing time of the data frame in SW, The time when the data frame is sent in SW, The time for data frames to queue in SW; The objective function is set to minimize the data frame transmission delay and minimize the data frame transmission jitter as shown in formula (2):

4. The method according to claim 3, characterized in that The method of inputting the data stream information to be processed in the TSN-5G heterogeneous network into the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm, wherein the train TSN-5G multi-agent reinforcement learning asynchronous scheduling algorithm takes each base station and switch as an independent agent, updates the state space and action space in the Markov model through iterative calculations, and outputs the scheduling scheme of the data stream to be processed, including: In the MADGNN algorithm of TSN-5G collaborative scheduling, BS and SW are used as intelligent agents. Each BS and SW has an independent actor network and critic network, which are deployed in the CNC server. BS obtains the current system's environmental state space through the Critic network. The environmental state space of the current system includes channel utilization, the number of data frame queues, and the wireless transmission time of data frames. The environmental state space of the current system is input into the Actor network of the BS. The BS generates an action through the Actor network. The action includes the transmission frequency and waiting time of the data stream. SW obtains the current device's environmental state space through the Critic network. The environmental state space of the current device includes the arrival time, compliance time, accumulated waiting time, and SW link utilization of the data frame. The environmental state space of the current device is input into the Actor network of the SW. The SW generates an action through the Actor network. The action includes the waiting time of the data stream. After BS and SW perform actions, the system feeds back reward values. The system stores the current state, action, reward, and next state into a four-tuple and stores it in the experience replay pool. When the number of samples in the replay pool reaches the threshold, the CNC server extracts samples from the replay pool and uses the four-tuple data in the samples to train and update the Actor network and Critic network.

5. The method according to claim 4, characterized in that The CNC server extracts samples from the playback pool and uses the quadruple data in the samples to train and update the Actor network and the Critic network, including: The CNC server extracts samples from the playback pool and transmits the four-tuple data in the samples to the Actor network and the Critic network. The Actor network and the Critic network adopt a dual network structure, using two forms: the current network and the target network. The Critic network uses the four-tuple data in the experience replay pool to calculate the evaluation value Q of the Actor network, and performs a gradient update based on the TD error, as shown in formula (6). The MSE loss function calculated by the TD temporal difference error is shown in formula (7). The parameters are updated using gradient descent In the above formula, Represents the parameters of the Critic network, expected value It is a four-tuple data sampled from the experience replay pool, including the current state s, the action a taken, the immediate reward r after execution, and the next state s after transfer ′ And whether it is a terminal state mark d, the current Q value function Estimate the value of taking action a in state s; The Actor network uses the quadruple data in the experience replay pool to calculate the MSE loss function of the TD error, as shown in formula (8), and calculates the MSE loss function about θ i The gradient of , using gradient descent to update the parameters, as shown in formula (9); θ i Represents the parameters of the Actor network, expected symbol Represents the expectation of extracting states from the experience replay pool D, which is used to estimate the average effect of the entire state space, the policy gradient Denotes the loss function for the policy parameter θ i The derivative of is used to indicate the optimization direction of the parameters. The gradient formula contains The term represents the rate of change of the Q-value function relative to the policy parameters, Represents the Actor network in a given state s i The output action below; The target network is updated by soft update, as shown in formula (10): represents the parameters of the main Q network, represents the parameters of the target Q network, θ represents the parameters of the main policy network, which is used to select actions, and θ ′ is the parameter of the target strategy network, and the parameter τ is the update coefficient, which is between 0 and 1 and controls the influence of the main network parameters on the target network parameters. In the update rule, The parameters of the main Q network and its original value Mixed in proportion τ and (1-τ), θ ′ Through the parameters θ of the main strategy network and its original value θ ′ Mix according to the proportion.

Images