Relay selection and resource allocation method for assisting inter-train communication based on wireless ad hoc network

Abstract

The invention discloses a relay selection and resource allocation method for assisting inter-train communication based on a wireless ad hoc network, and relates to an optimization method for relay selection and resource allocation of the inter-train communication assisted by the wireless ad hoc network in a communication-based train control (CBTC) system. In order to ensure the transmission quality of a wireless network and reduce the deployment cost, a wireless ad hoc network is applied to a CBTC system. In order to further improve the transmission efficiency of the multi-hop ad hoc network, a joint optimization problem of link throughput and total transmission consumption energy is modeled as a Markov decision process. Because the transmission link between the trains is in a distributed state, a federated soft actor-commentator algorithm is designed for solving, so that each agent can synchronize the dynamic state of the whole network. Simulation results show that the throughput of the inter-train transmission link and the energy consumed by transmission in the method are superior to those in other methods.

Description

Technical Field

[0001] The present invention relates to an optimization method for relay selection and resource allocation of wireless ad hoc network-assisted train-to-train communication in a communication-based train control (CBTC) system, belonging to the technical field of communication networks. Background Art

[0002] With the continuous improvement of wireless mobile technology, the application requirements for mobile devices are getting higher and higher. Connecting mobile terminals to wireless networks for information exchange has enriched the functions of mobile devices and derived a series of applications based on Internet of Things technology. The communication-based train control (CBTC) system is one of these applications. The train accesses the network through the on-board transmission unit and obtains the speed and position information of the train ahead to achieve safety detection. At the same time, other applications based on train communication, such as driver assistance, user entertainment and safety protection, are gradually being improved.

[0003] To ensure high reliability of message transmission, it is crucial to choose a suitable mobile communication network. As one of the common communication networks, the long-term evolution (LTE) network requires the support of fixed infrastructure such as roadside base stations and optical cables, which greatly increases the cost of network deployment and maintenance. In addition, the coverage of base stations is limited, making it difficult to achieve continuous connection of mobile devices. Therefore, traditional networks cannot meet the transmission needs of highly mobile trains. A wireless ad hoc network is composed of a group of wireless terminals with transceivers. The network does not require fixed infrastructure and can be organized flexibly. Mobile ad hoc network nodes can freely join and leave the network to adapt to changes in complex environments. At the same time, ad hoc networks can solve the problem of long signal delays or interruptions caused by frequent switching between base stations in wireless local area networks (WLAN) and LTE networks. Although wireless ad hoc networks are rarely used in CBTC systems, trains can establish fast connections with roadside nodes through their own signal transmitters to reduce delays and interruptions caused by switching. Because of these advantages, wireless ad hoc networks can better meet the needs of CBTC systems.

[0004] However, there are still some problems to be improved in wireless ad hoc assisted rail transit communication. During the operation of trains, high-speed trains lead to rapid changes in the topology of wireless ad hoc networks, which reduces the channel quality and transmission throughput of train-to-train (T2T) and train-to-wayside (T2W) links. At the same time, a large amount of real-time data is transmitted between trains, such as safety information and user entertainment information. In order to reduce the delay and congestion of data transmission in wireless ad hoc networks, the throughput should be considered and optimized. However, if the throughput is blindly increased while ignoring the energy overhead of the network, it will lead to excessive energy consumption. Therefore, it is crucial to reduce energy consumption while ensuring transmission quality.

[0005] In order to achieve the optimization goals of improving throughput and reducing energy consumption, it is necessary to reasonably allocate limited relays, subchannels and energy resources. As one of the deep reinforcement learning (DRL) algorithms, the soft actor-critic (SAC) algorithm can derive appropriate strategies based on the time-varying state of the environment. The algorithm combines maximum entropy and offline strategies to improve the exploration ability of the model and avoid the agent from falling into the local optimum. However, in the CBTC system, a large amount of data exchange during centralized training will lead to high latency in real-time applications. In order to speed up the convergence of the model, the SAC algorithm is combined with federated learning, and the federated soft actor-critic (FeSAC) algorithm is proposed. In this way, the communication overhead caused by direct information exchange can be significantly reduced, while also ensuring the data security of the train.

[0006] In order to optimize the link transmission quality of wireless ad hoc networks in CBTC systems, this paper develops a new multi-hop ad hoc network transmission framework based on the FeSAC algorithm to address the issues of channel allocation, multi-hop task forwarding, and power selection in rail transit scenarios. This framework comprehensively considers factors such as relay selection, sub-channel selection, and train transmission power, and reasonably allocates transmission resources between trains with the goal of maximizing T2T link throughput while minimizing transmission energy consumption. Summary of the invention

[0007] In order to further improve the transmission efficiency of multi-hop ad hoc networks, this method models the joint optimization problem of link throughput and total transmission energy consumption as a Markov decision process (MDP). Since the transmission links between trains are in a distributed state, a federated soft actor-critic (FeSAC) algorithm is designed to solve the problem by combining federated learning with the soft actor-critic (SAC) algorithm, so that each agent can synchronize the dynamics of the entire network.

[0008] The main purpose of the present invention is to comprehensively consider the link reliability, interference between multiple transmission links and relay packet loss rate when multiple groups of trains communicate in the scenario of high-speed train movement. Under the condition of ensuring the link reliability and queuing delay requirements of T2T communication, the link throughput is maximized while consuming the least energy. In order to achieve this goal, the newly proposed FeSAC algorithm is used to train the model to obtain the optimal trackside relay selection and network resource allocation scheme. This method solves the problem of how to determine the optimal wireless ad hoc network relay selection, subchannel allocation and transmission power allocation in the scenario of multiple trains communicating at the same time. And by executing the optimal network resource allocation strategy, the throughput is effectively increased and energy consumption is reduced.

[0009] The scenario model diagram of the wireless ad hoc network assisted train-to-train communication adapted by the present invention is shown in Figure 1 .

[0010] The system operation principle flow chart of the technical solution of the present invention is shown in Figure 2 .

[0011] The relationship between the throughput of the system of the present invention and the distance between two communication trains is shown in Figure 3 .

[0012] The transmission energy consumption of the present invention and the distance between two communication trains are shown in FIG. Figure 4 .

[0013] The relationship diagram between the link throughput of the system of the present invention and the relay node queuing delay threshold is shown in Figure 5 .

[0014] The relationship between the throughput of the system of the present invention and the number of neighbor links communicating at the same time is shown in Figure 6 .

[0015] The relationship between the energy consumption of the system of the present invention and the number of neighbor links communicating at the same time is shown in Figure 7 .

[0016] The scenario model of the multi-hop self-organizing network auxiliary train and train communication of the present invention is as follows Figure 1As shown in the figure, at a certain moment there are L transmitting links and R fixed trackside MANET nodes. When the number of trains and trackside MANET nodes is known, the CBTC system model is set according to the actual scenario, and the weight factors of throughput and energy consumption are determined. Then, the relay node selection, subchannel allocation and transmission power setting are comprehensively considered, and the system model optimization problem is constructed in combination with the system scenario and optimization objectives. Finally, the FeSAC optimization algorithm is used to solve it in a distributed manner, so as to achieve the goal of minimizing energy consumption while maximizing system throughput. Specifically, follow the following steps in sequence:

[0017] In step (1), we consider the deployment of trackside equipment with ad hoc network nodes along the track to provide communication services for trains. The sets of T2T links and trackside nodes in transmission are L = {1,2…l…L} and R = {1,2…r…R}, indicating that there are L T2T links in communication and R trackside nodes providing services. At the same time, the entire system uses OFDM technology, and the set of subchannels is represented by F = {1,2…f,…,F}. Since the transmission distance between trains may be greater than the single-hop distance of the ad hoc network, trackside nodes are required to assist in transmission. In order to ensure the reliability of transmission of each node on the train, multiple relay nodes are set up in the train coverage area.

[0018] Step (2) calculates the minimum signal-to-noise ratio threshold that the communication link needs to meet when the two trains communicate. During the operation of the train, as the train accelerates and decelerates, the distance between the two trains will change. Since the front and rear trains are both running at high speed, the relative speed has a great impact on the distance. Therefore, the Wiener process with drift parameters can better characterize the dynamic changes in the distance between the trains. According to the Wiener process, the speed change of the train during the movement can be expressed as

[0019]

[0020] in and is the speed of train s at time slots t1 and t2, and Δt is t2-t1. s is the acceleration of the train in Δt, γ s Obeys the standard normal distribution. Therefore, the distance change between train s and train d is

[0021]

[0022] in and is the speed change of train s and train d in time Δt.

[0023] When two trains are moving in the same direction on the same track, the speed of the front train d is greater than the speed of the rear train s. After Δt time, the distance between the two trains can be expressed as

[0024]

[0025] The current vehicle speed d is less than the following vehicle speed s, that is, v d <v s When , the distance between them after Δt can be expressed as

[0026]

[0027] After Δt, if the train is still within the communication range, the link connection is stable. When the signal-to-noise ratio of the link directly connected between trains is less than the specified threshold, the link is disconnected. In order to ensure the reliability of the link connection, the interruption probability Y(l sd ) must be less than the interruption probability P th As the distance between two trains increases, the probability of interruption increases. The reliability requirement of the link between trains can be expressed as

[0028] Y(l sd )=P{γ sd <γ0} <P th

[0029] The link signal-to-noise ratio is expressed as

[0030]

[0031] Among them, P s is the transmission power of train s, I(d,t2) is the interference received by train d at time t2, h(l sd ,t2) represents the transmission loss between train s and train d, which can be calculated as

[0032]

[0033] where X(sd,Δt) is the fading coefficient, β and represents the path loss index and the distance between the two trains. Therefore, the communication signal-to-noise ratio γ between the two trains sd The range can be expressed as

[0034]

[0035] Step (3), when the link meets the minimum signal-to-noise ratio threshold, trains can communicate directly with each other, and the throughput of the communication link is calculated. When two trains communicate, if they choose the same sub-link as other communication links, interference will occur between the links, which will affect the throughput of the link. Therefore, the signal-to-noise ratio of the communication link between train s and train d is

[0036]

[0037] where σ 2 is the power of the Gaussian noise, is the interference caused by other links l using the same subchannel. Indicates that link l is using the same subchannel for transmission. In the process of train-to-train communication, the throughput of the direct transmission link TH sd Expressed as

[0038] TH sd =R sd =Blog(1+γ sd ).

[0039] Where B is the bandwidth of the subchannel.

[0040] Step (4), when the inter-train transmission link does not meet the minimum signal-to-noise ratio threshold, calculate the link throughput using relay transmission. When relay r is used to assist transmission, the signal-to-noise ratio of the communication link between train s and train d can be expressed as

[0041]

[0042] The corresponding throughput is expressed as

[0043]

[0044] Where R sr is the throughput between train s and relay r, R rd is the throughput between relay r and train d.

[0045] However, when the relay node has a large number of packets to transmit, the queuing delay will increase. Greater than the threshold D th , then the transmission fails and the data packet will be discarded with a probability of being discarded.

[0046]

[0047] in, is the link throughput of the train-to-train communication using relay r. th and is the queue delay threshold and the average number of packages in the queue, l ave is the average size of the queued packets. Therefore, the effective throughput of using relay nodes for transmission is

[0048]

[0049] Step (5) calculates the energy consumed when the trains transmit directly or use relay transmission. When the package is transmitted directly, only the energy consumption of train s is included. However, when the relay-assisted train is transmitted, the energy consumed includes the energy consumed by train s and relay r.

[0050] In order to calculate the energy consumed by communication, the transmission delay is calculated first. Therefore, the transmission delay of direct communication between trains is expressed as

[0051]

[0052] Where Q t is the size of the packet transmitted in time slot t, TH sd is the effective throughput of the transmission link. When the trackside relay assists the inter-train communication, the communication delay is expressed as

[0053]

[0054] In relay-assisted transmission, we set the transmission power of the source train Transmission power of the relay The same, so the transmission power of the entire link can be expressed as Therefore, the energy consumption of train-to-train communication can be calculated as

[0055]

[0056] Step (6), based on steps (1)-(5), combine the scenario and optimization objectives to construct a model optimization problem.

[0057] The optimization objective is expressed as

[0058]

[0059] stC1:Y(l sd ) <P th #

[0060]

[0061] Among them, β and (1-β) are the weights of link throughput and energy consumption. Constraint 1 ensures that the interruption probability of the transmission link meets the reliability range. Constraint 2 means that during the transmission process, the packet loss rate of the relay node needs to be less than the set threshold. Constraint 3 means that each transmission link needs to select a subchannel for transmission. Constraint 4 is the transmission power range of the train.

[0062] Step (7), according to step (6), the joint optimization problem is transformed. The specific steps are as follows:

[0063] Step (7.1), the state space defined by the present invention consists of five key variables, namely, the distance between two trains d(t), the distance between the train and the relay r(t), the number of queued packages for each relay q(t), the situation of the subchannels occupied by the communication link between adjacent trains n(t) and the transmission power p(t) of other adjacent trains. Therefore, the state space is set as:

[0064] s(t)={d(t),r(t),q(t),n(t),p(t)}

[0065] Among them, the distance d(t) between the two trains is designed to be between 850m and 1450m, the distance r(t) between the train and the relay is calculated according to the coordinates, the queued packages q(t) of each relay is between 2 and 10, the sub-channel occupancy is represented by n(t) and the transmission power p(t) of the adjacent train is between 13dBm and 20dBm.

[0066] In step (7.2), in order to obtain better long-term returns, it is necessary to flexibly adjust the action strategy according to environmental changes. Therefore, the action space is set as follows:

[0067] a(t)={R(t),C(t),P(t)}

[0068] Where R(t) = {0, 1, ... r ... R}, if R(t) = 0, it means that the data packet is transmitted directly between the two trains. In addition, R(t) represents the number of relays passed during transmission. C(t) ∈ {1, 2 ... f, ..., F} represents the number of selected subchannels. P(t) ∈ [13, 20] dBm represents the transmission power of the train.

[0069] In step (7.3), in order to jointly consider the throughput and consumed energy, the reward function is defined as the weighted sum of throughput and energy. At the same time, the interruption probability of the transmission link between trains should meet the condition of C1, and the packet loss rate should also meet the condition of C2. Otherwise, it is considered that the package transmission fails, corresponding to the corresponding penalty value δ. The formula is:

[0070]

[0071] Among them, TH sd represents the link throughput when train s and train d are transmitting, E s Represents the energy consumption of train transmission.

[0072] Step (8), according to the state space, action space and reward function constructed in step (7), complete the setting of parameters such as the execution strategy of the deep reinforcement learning algorithm FeSAC and the neural network training.

[0073] Since the transmission power selection of the train is a continuous action, the SAC algorithm proposed for the continuous state space is used to solve this problem. Unlike continuous space action selection algorithms such as DDPG, the SAC algorithm adopts the maximum entropy strategy, which increases the exploration ability of the agent's action selection and prevents the agent from falling into the local optimum. In addition, the transmission link between trains is distributed, and each train makes a separate decision as an agent. In order to avoid the problem of insufficient network training due to the small amount of data of a single agent, a federated learning method is adopted to integrate the training parameters of L agents to make the agent converge faster. The design and training of the FeSAC algorithm model proposed in the present invention are as follows.

[0074] In step (8.1), each agent is first trained locally. Each train-to-train transmission link can be regarded as an independent SAC agent. Each SAC agent contains an actor network and two critic networks. The critic network evaluates the action value according to the current state. t ,a t ), while the actor network infers the probability distribution of consecutive actions.

[0075] In order to enhance the randomness and exploration of action selection, the algorithm also increases information entropy. By training the deep reinforcement learning network, the optimal strategy π(s t ,a t ) to maximize the long-term reward of each agent. Therefore, the optimization goal of the SAC algorithm is:

[0076]

[0077] Among them, π(·|s t ) represents the probability distribution of the mapping from state space to action space. logπ(·|s t ) represents the entropy of the policy. α is a parameter that weighs entropy and reward. γ∈[0,1] represents the importance parameter between current and future reward values.

[0078] In order to obtain the optimal strategy, the SAC algorithm uses the gradient descent method to update the neural network. The SAC algorithm contains two groups of critic networks with the same neural network structure. Each group of critic networks contains an evaluation network with a parameter θ and a parameter The target network consists of , and the soft Bellman residual is used to update the parameters of each critic network, which is expressed as:

[0079]

[0080] Where D is a batch of training experiences stored in the replay buffer.

[0081] After a period of training, the parameters of the key evaluation network are used to soft-update the key target network. The update process can be expressed as

[0082]

[0083] Where τ is the ratio of soft updates.

[0084] Furthermore, the actor network is trained by minimizing the Kullback-Leibler divergence, which can be expressed as

[0085]

[0086] in, Represents a neural network transformation to reparameterize the strategy. ∈ n is the input noise vector.

[0087] Step (8.2), then perform parameter aggregation for multiple agents. In each round of training, each agent uploads its own parameters to the global network, and then allocates different weights for aggregation according to the reward value of the individual agent training. Each SAC network consists of five networks, including The performer network, two critic-evaluator networks with parameters θ and two The parameters of the global actor network can be calculated as

[0088]

[0089] in, is the reward value obtained by the lth agent in the jth round of training, and is the total reward value of all agents, are the parameters of the lth actor network. Similarly, the global critic-critic network and the critic-target network are aggregated in the same way.

[0090] Step (8.3) aggregates the model parameters of multiple agents. The aggregated weights are distributed to each agent, and each agent uses the existing weights for the next round of training.

[0091] Step (9), according to the deep neural network trained in step (8), the optimal strategy for the optional actions in each state is obtained, the action generated by the strategy is used as the optimal action in the state, and the optimal action in each state is continuously executed until the execution of the instruction is completed.

[0092] The advantage of the present invention is that, in high mobility and complex multi-train communication scenarios, for a multi-hop ad hoc network-assisted CBTC system, the selection of trackside relay nodes, subchannel selection and transmission power selection are comprehensively considered, and energy consumption is reduced while ensuring the train-to-train transmission throughput. The influence of the relay selection and resource allocation algorithm based on multi-hop ad hoc network-assisted train-to-train transmission on the system throughput and energy consumption in the optimization scenario is examined through simulation experiments. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 ,The communication scenario includes a schematic diagram of the network model of wireless ad hoc network-assisted train-to-train communication.

[0094] Figure 2 , Flowchart of the urban rail relay selection and resource optimization method based on wireless ad hoc network-assisted train-to-train communication.

[0095] Figure 3 , the relationship between the distance between two trains and the throughput under different algorithms. Among them, the diamond represents the method optimized under the FeSAC algorithm of the present invention, the pentagon represents the method optimized under the SAC algorithm, the square represents the method optimized under the DDPG algorithm, the triangle represents the comparison method under the random selection of subchannels, and the circle represents the comparison method under the random selection of relays.

[0096] Figure 4 , the relationship between the energy consumption of train link transmission and the distance between two trains. Among them, the diamond represents the method optimized under the FeSAC algorithm of the present invention, the pentagon represents the method optimized under the SAC algorithm, the square represents the method optimized under the DDPG algorithm, the triangle represents the comparison method under the random selection of sub-channels, and the circle represents the comparison method under the random selection of relays.

[0097] Figure 5 , which is a relationship diagram between link throughput and queuing delay threshold. Among them, the diamond represents the method optimized under the FeSAC algorithm of the present invention, the pentagon represents the method optimized under the SAC algorithm, the square represents the method optimized under the DDPG algorithm, the triangle represents the comparison method under the random selection of sub-channels, and the circle represents the comparison method under the random selection of relays.

[0098] Figure 6 , which is a graph showing the relationship between link throughput and the number of transmission links between adjacent trains. Among them, the diamond represents the method optimized under the FeSAC algorithm of the present invention, the pentagon represents the method optimized under the SAC algorithm, the square represents the method optimized under the DDPG algorithm, and the triangle represents the comparison method under the random selection of sub-channels.

[0099] Figure 7, which is a graph showing the relationship between link energy consumption and the number of transmission links between adjacent trains. Among them, the diamond represents the method optimized under the FeSAC algorithm of the present invention, the pentagon represents the method optimized under the SAC algorithm, the square represents the method optimized under the DDPG algorithm, and the triangle represents the comparison method under the random selection of sub-channels. DETAILED DESCRIPTION

[0100] The following is a further explanation of the technical solution of relay selection and resource allocation optimization based on wireless ad hoc network-assisted inter-train communication in conjunction with the accompanying drawings and examples.

[0101] The method flow chart of the present invention is as follows Figure 2 As shown, the following steps are included:

[0102] Step 1: Initialize the multi-hop wireless ad hoc network-assisted CBTC system and set the number of trackside relay nodes and communicating trains.

[0103] Step 2: According to the initialization conditions and actual conditions, calculate the link transmission threshold γ that needs to be met between the two trains sd and TH for direct transmission between two trains sd If the link of direct transmission between trains does not meet the threshold condition, then the information transmission between trains is assisted by relays, and the link throughput using relay transmission is calculated. Finally, calculate the energy consumption E of the transmission link s .

[0104] Step 3: Based on the link transmission throughput TH between trains sd and energy consumption E s ,Combining the scenario and optimization objectives, a system model joint optimization problem is constructed.

[0105] Step 4: Establish the optimization problem and model the Markov decision process, setting the state space, action space and reward function;

[0106] Step 5: Complete the setting of parameters such as algorithm execution strategy and neural network training according to the FeSAC algorithm;

[0107] Step 6: According to different states, the central agent integrates the training parameters in each agent network to obtain the optimal action decision so that the agent can obtain the maximum benefit.

[0108] Figure 3is a graph showing the relationship between the distance between the two trains and the throughput. It can be noted that when the two trains are far apart, the throughput of the link decreases. The reason is that the distance between the two trains affects the size of the signal-to-noise ratio. As the distance increases, the signal-to-noise ratio decreases. At the same time, due to the long distance, direct communication between trains becomes difficult, so relay-assisted transmission is more likely to be selected. However, if the queuing delay exceeds the threshold, the package will be discarded, which will also lead to a decrease in effective throughput. In addition, the FeSAC algorithm outperforms other schemes in improving throughput, indicating that the FeSAC algorithm can be used to optimize relay selection and sub-channel selection. Simulation results show that the FeSAC algorithm can select sub-channels with less interference, or select relays with shorter queuing delays.

[0109] Figure 4 This is a graph showing the relationship between the distance between two trains and the transmission energy consumption. It can be seen that as the distance increases, the energy consumption increases monotonically. In order to meet the transmission interruption probability between trains, as the distance between the two trains increases, the transmission power required by the trains gradually increases, so the energy consumption of information transmission also increases. Compared with deep learning algorithms such as DDPG and SAC, the FeSAC algorithm selects relay nodes, subchannels, and transmission power to minimize the energy consumed by the transmission link when the distance between the two trains is the same.

[0110] Figure 5 The figure shows the relationship between link throughput and relay node queuing delay threshold. It can be seen that the throughput of all algorithms increases slightly with the increase of delay threshold, which means that when the delay threshold increases, the probability of packet loss decreases, so the effective throughput of the link increases. Under the same delay threshold, the FeSAC algorithm is always better than the DDPG and SAC algorithms. This shows that the FeSAC algorithm can better select appropriate transmission relays and subchannels.

[0111] Figure 6 is a graph showing the relationship between link throughput and the number of transmission links between adjacent trains. During the peak period of train operation, the number of communication links occupying the same subchannel gradually increases. It can be seen that as the number of adjacent trains increases, the throughput of the optimized inter-train communication link tends to decrease. The reason is that as the number of links increases, the mutual interference within the same channel also intensifies. This interference effect directly affects the quality of communication between trains, which in turn leads to a decrease in throughput. In addition, compared with the case of randomly selecting spectrum resources, the FeSAC algorithm is able to select subchannels that are less occupied by adjacent links for communication, thereby effectively reducing the interference caused by other links.

[0112] Figure 7Figure 2 is a graph showing the relationship between link energy consumption and the number of transmission links between adjacent trains. As the number of links between adjacent trains increases, the interference to the optimized links between trains increases. In order to make the link interruption probability less than the threshold in constraint C1, the transmission power needs to be increased to maintain the link signal-to-noise ratio, thereby increasing the energy overhead. Compared with other algorithms, when the number of links between adjacent trains increases, the FeSAC algorithm can still select channels with less interference and select appropriate power for transmission, thereby reducing energy consumption.

Claims

1. A method for relay selection and resource allocation based on wireless ad hoc network-assisted inter-train communication, characterized in that: The following steps are involved: Step 1: Initialize the multi-hop MANET-assisted CBTC system and set the number of trackside nodes and the number of inter-train links being transmitted; Step 2: Calculate the minimum signal-to-noise ratio threshold that the train-to-train communication link needs to meet based on the initialization conditions and actual conditions; Step 3: According to the initialization conditions and actual conditions, calculate the link throughput TH of direct communication between trains when the link meets the minimum signal-to-noise ratio threshold. sd ; Step 4: Calculate the link throughput using relay-assisted transmission when the link signal-to-noise ratio is lower than the minimum threshold based on the initialization conditions and actual conditions. Step 5: Calculate the energy consumed by direct transmission or relay-assisted transmission based on the two transmission conditions in step 3 and step 4. s ; Step 6: Combine the scenario and optimization goal to build a system model joint optimization problem; Step 7: Convert the joint optimization problem into a Markov decision process, set the state space, action space and reward function; Step 8: According to the proposed state space, action space and reward function, complete the setting of the deep reinforcement learning algorithm FeSAC execution strategy and neural network training parameters; Step nine: Select the optimal action based on the optimal strategy obtained in each state to obtain the maximum benefit.

2. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 1, characterized in that: In step 1, communication services are provided for the train. The sets of T2T links and trackside nodes that are transmitting are L = {1,2…l…L} and R = {1,2…r…R}, indicating that there are L communicating T2T links and R trackside nodes providing services. Using OFDM technology, the set of subchannels is represented by F = {1,2…f,…,F}. Trackside nodes are used to assist transmission, and multiple relay nodes are set up in the train coverage area to assist transmission.

3. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 2, characterized in that: In step 2, the minimum signal-to-noise ratio threshold that the communication link between the two trains needs to meet is calculated; during the operation of the train, as the train accelerates and decelerates, the distance between the two trains will change; the Wiener process with drift parameters can better characterize the dynamic changes in the distance between the trains; according to the Wiener process, the speed change of the train during the movement is expressed as: in and is the speed of train s at time slots t1 and t2, Δt is t2-t1; a s is the acceleration of the train in Δt, γ s It obeys the standard normal distribution; therefore, the distance between train s and train d changes as follows: in and is the speed change of train s and train d in time Δt; When two trains are moving in the same direction on the same track, the speed of the front train d is greater than the speed of the rear train s; after Δt time, the distance between the two trains is expressed as: The current vehicle speed d is less than the following vehicle speed s, that is, v d <v s When , the distance after Δt is expressed as: After Δt time, if the train is still within the communication range, the link connection is stable; when the signal-to-noise ratio of the link directly connected between trains is less than the specified threshold, the link is interrupted; to ensure the reliability of the link connection, the interruption probability Y(l sd ) must be less than the interruption probability P th ; As the distance between two trains increases, the probability of interruption increases; the reliability requirement of the link between trains is expressed as: Y(l sd )=P{γ sd <γ0} <P th The link signal-to-noise ratio is expressed as: Among them, P s is the transmission power of train s, I(d,t2) is the interference received by train d at time t2, h(l sd , t2) represents the transmission loss between train s and train d, which is calculated as: where X(sd,Δt) is the fading coefficient, β and represents the path loss index and the distance between the two trains; therefore, the communication signal-to-noise ratio γ between the two trains sd The range is expressed as:

4. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 3, characterized in that: In step 3, when the link meets the minimum signal-to-noise ratio threshold, the link throughput of direct communication between trains is calculated; when two trains communicate, if the same sub-link as other communication links is selected, interference will occur between the links, thereby affecting the link throughput; the signal-to-noise ratio of the communication link between train s and train d is: where σ 2 is the power of the Gaussian noise, is the interference caused by other links l using the same subchannel; where, Indicates that link l is using the same subchannel for transmission; during the communication between trains, the throughput of the direct transmission link TH sd It is expressed as: TH sd =R sd =Blog(1+c sd ). Where B is the bandwidth of the subchannel.

5. The method for relay selection and resource allocation based on wireless ad hoc network-assisted inter-train communication according to claim 4, characterized in that: In step 4, when the inter-train transmission link does not meet the minimum signal-to-noise ratio threshold, the link throughput using relay transmission is calculated; when relay r is used to assist transmission, the signal-to-noise ratio of the communication link between train s and train d is expressed as: The corresponding throughput is expressed as: Where R sr is the throughput between train s and relay r, R rd is the throughput between relay r and train d; However, when the relay node has a large number of packets to transmit, the queuing delay will increase; if the queuing delay Greater than the threshold D th , then the transmission fails and the data packet will be discarded. The probability of discarding for: in, is the link throughput of the train-to-train communication using relay r; D th and is the queue delay threshold and the average number of packages in the queue, l ave is the average size of queued packets; the effective throughput using relay nodes is:

6. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 5, characterized in that: In step 5, the energy consumed when the trains transmit directly or use relay transmission is calculated; when the package is transmitted directly, only the energy consumption of train s is included; however, when the relay-assisted train is transmitted, the energy consumed includes the energy consumed by train s and relay r; In order to calculate the energy consumed by communication, the transmission delay is calculated first; the transmission delay of direct communication between trains is expressed as: Where Q t is the size of the packet transmitted in time slot t, TH sd is the effective throughput of the transmission link; when the trackside relay assists the inter-train communication, the communication delay is expressed as: In relay-assisted transmission, set the transmission power of the source train Transmission power of the relay The transmission power of the entire link is expressed as Therefore, the energy consumption of train-to-train communication is calculated as:

7. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 6, characterized in that: In step 6, based on steps 1 to 5, the model optimization problem is constructed by combining the scenario and the optimization goal; the optimization goal is expressed as: Among them, β and (1-β) are the weights of link throughput and energy consumption; Constraint 1 ensures that the interruption probability of the transmission link meets the reliability range; Constraint 2 means that during the transmission process, the packet loss rate of the relay node needs to be less than the set threshold; Constraint 3 means that each transmission link needs to select a subchannel for transmission; Constraint 4 is the transmission power range of the train.

8. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 7, characterized in that: In step 7, the joint optimization problem is transformed according to step 6. The specific steps are as follows: In step (7.1), the state space defined consists of five key variables, namely, the distance between the two trains d(t), the distance between the train and the relay r(t), the number of queued packages for each relay q(t), the subchannel occupied by the communication link between adjacent trains n(t), and the transmission power p(t) of other adjacent trains; therefore, the state space is set as: s(t)={d(t),r(t),q(t),n(t),p(t)} The distance d(t) between two trains is designed to be between 850m and 1450m, the distance r(t) between the train and the relay is calculated according to the coordinates, the number of queued packages q(t) for each relay is between 2 and 10, the subchannel occupancy is represented by n(t) and the transmission power p(t) of the adjacent train is between 13dBm and 20dBm; Step (7.2), in order to obtain better long-term returns, it is necessary to flexibly adjust the action strategy according to environmental changes; set the action space as follows: a(t)={R(t),C(t),P(t)} Where R(t) = {0, 1, ... r ... R}, if R(t) = 0, it means that the data packet is transmitted directly between the two trains; in addition, R(t) represents the number of relays passed during transmission; C(t) ∈ {1, 2 ... f, ..., F} represents the number of selected subchannels; P(t) ∈ [13, 20] dBm represents the transmission power of the train; In step (7.3), in order to jointly consider the throughput and the consumed energy, the reward function is defined as the weighted sum of the throughput and the energy. At the same time, the interruption probability of the transmission link between trains should meet the condition of C1, and the packet loss rate should also meet the condition of C2; otherwise, it is regarded as a package transmission failure, corresponding to the corresponding penalty value δ; its formula is: Among them, TH sd represents the link throughput when train s and train d are transmitting, E s Represents the energy consumption of train transmission.

9. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 8, characterized in that: In step 8, according to the state space, action space and reward function constructed in step 7, the parameters such as the execution strategy of the deep reinforcement learning algorithm FeSAC and the neural network training are set; Step (8.1), first each agent performs local training, each train and train transmission link is regarded as an independent SAC agent, each SAC agent contains an actor network and two critic networks; the critic network evaluates the action value according to the current state Q(s t ,a t ), while the actor network infers the probability distribution of consecutive actions; In order to enhance the randomness and exploration of action selection, the SAC algorithm increases information entropy; by training the deep reinforcement learning network, the optimal strategy π(s t ,a t ) to maximize the long-term reward of each agent; therefore, the optimization goal of the SAC algorithm is: Among them, π(·|s t ) represents the probability distribution of the mapping from state space to action space; logπ(·|s t ) represents the entropy of the strategy; α is a parameter that weighs entropy and reward; γ∈[0,1] represents the importance parameter between current and future reward values; To obtain the optimal strategy, the SAC algorithm uses the gradient descent method to update the neural network. The SAC algorithm includes two groups of critic networks with the same neural network structure. Each group of critic networks includes an evaluation network with a parameter θ and a parameter The target network consists of , and the soft Bellman residual is used to update the parameters of each critic network, which is expressed as: Where D is a batch of training experiences stored in the replay buffer; After a period of training, the parameters of the key evaluation network are used to soft-update the key target network; the update process is expressed as: Where τ is the ratio of soft updates; Furthermore, the actor network is trained by minimizing the Kullback-Leibler divergence, expressed as: in, Represents the reparameterization of the strategy by neural network transformation; ∈ n is the input noise vector; Step (8.2) aggregates the parameters of multiple agents. In each round of training, each agent uploads its own parameters to the global network, and according to the reward value of the training of a single agent, different weights are assigned for aggregation. Each SAC network contains five networks, including The performer network, two critic-evaluator networks with parameters θ and two The critic-target network of ; where the parameters of the global actor network are calculated as: in, is the reward value obtained by the lth agent in the jth round of training, and is the total reward value of all agents, are the parameters of the lth actor network; the global critic-critic network and the critic-target network are aggregated in the same way; Step (8.3) aggregates the model parameters of multiple agents; the aggregated weights are distributed to each agent, and each agent uses the existing weights for the next round of training.

10. The method for relay selection and resource allocation based on wireless ad hoc network-assisted train-to-train communication according to claim 9, characterized in that: In step nine, the optimal strategy for optional actions in each state is obtained based on the deep neural network trained in step eight, the action generated by the strategy is used as the optimal action in that state, and the optimal action in each state is continuously executed until the execution of the instruction is completed.

Images