Train handover control method based on prediction enhancement

Abstract

The invention provides a train handover control method based on prediction enhancement. The method comprises the following steps: training a wireless communication quality parameter prediction model by using a time sequence data set to obtain a trained multivariable time sequence prediction model, predicting time sequence data of a time point of a future time step length, and generating state data of a future time window; the method comprises the following steps: constructing an actor-commentator framework comprising a strategy network and a value network, iteratively training a deep reinforcement learning decision model by utilizing the actor-commentator framework based on historically collected experience training data, updating parameters of the strategy network and the value network by adopting a near-end strategy optimization algorithm through minimizing a composite loss function, and obtaining the value of the strategy network and the value network. The training is converged, and a trained deep reinforcement learning decision model is obtained; and the trained deep reinforcement learning decision model is utilized to control the train to execute a handover action in actual operation. The stability of the train system is improved, and the communication service quality under high-speed movement of the train is guaranteed.

Description

Technical Field

[0001] This invention relates to the field of train operation control technology, and in particular to a train cross-zone handover control method based on predictive enhancement. Background Technology

[0002] Autonomous train operation systems represent a significant development direction for urban rail transit systems. These systems rely on the train's own resource management and real-time positioning to calculate operating permits, thereby simplifying trackside equipment, shortening headways, and improving track resource utilization. Because autonomous train operation systems heavily depend on real-time data exchange and stable communication links, they place higher demands on the quality of service (QoS) of train-to-ground wireless communication.

[0003] In vehicle-to-ground communication networks, base stations are deployed in a chain topology along the track to provide continuous coverage. Mobility management via handover is crucial for maintaining service continuity and quality of service (QoS) of wireless communication as trains move between base stations. However, inappropriate base station handover triggering can lead to core network signaling burden and even wireless connection failures. Ping-pong handover, where trains rapidly switch back and forth between two adjacent base stations, is not only inefficient and consumes significant network resources, but also significantly increases the probability of wireless connection failures. The resulting connection interruptions degrade QoS of wireless communication and pose a safety risk to autonomous train operation systems that rely on real-time control data.

[0004] To address the challenges of handover in high-speed moving scenarios, academia and industry have conducted extensive research. Currently, one existing train handover control method includes short-sighted decision-making and oscillatory decision-making. This method utilizes deep reinforcement learning to learn adaptive policies from interactions with the environment. However, this method has drawbacks. In highly mobile and non-stationary scenarios such as vehicle-to-ground wireless communication, relying solely on historical data and instantaneous observations of state representations can easily lead to short-sighted decisions and oscillatory behavior by the agent.

[0005] Another existing train handover control method includes a non-end-to-end control method. This method utilizes channel prediction to provide the system with forward-looking decision-making capabilities, enabling it to anticipate link quality degradation. The drawbacks of this method include the lack of an integrated, end-to-end decision-making framework; its operational logic is typically limited to predefined, event-based triggers, hindering the autonomous learning of dynamic control strategies. Summary of the Invention

[0006] Embodiments of the present invention provide a train handover control method based on prediction enhancement to ensure the quality of communication services when trains are moving at high speeds.

[0007] To achieve the above objectives, the present invention adopts the following technical solution.

[0008] A train handover control method based on prediction enhancement includes: In response to the characteristics of train operation in urban rail transit scenarios, wireless communication quality parameters and train operation status information of multiple cells are periodically collected during the train's journey, and the collected data are used to construct a time-series dataset. A wireless communication quality parameter prediction model is trained using the aforementioned time series dataset to obtain a trained multivariate time series prediction model. This trained model is then used to generate future... Time series data at time points with time steps; Historical wireless communication quality parameters from urban rail transit scenarios are stitched together to obtain a linear extrapolation of historical observations. A confidence-weighted mechanism is then used to correlate this linear extrapolation with subsequent future data. Convex combination of time series data at time points of length is used to generate state data for future time windows; An actor-critic framework comprising a policy network and a value network is constructed. Based on historically collected experience training data, a deep reinforcement learning decision model is iteratively trained using the actor-critic framework. A proximal policy optimization algorithm is employed to update the parameters of the policy network and the value network by minimizing the composite loss function until training converges, resulting in a well-trained deep reinforcement learning decision model. The well-trained deep reinforcement learning decision model is then used to control the train to perform cross-zone handover actions during actual operation.

[0009] Preferably, the method of periodically collecting wireless communication quality parameters and train operation status information of multiple cells during train travel, based on the characteristics of train operation in urban rail transit scenarios, and using the collected data to construct a time-series dataset, includes: In a fixed scenario where multiple base stations are linearly deployed along a track, the wireless communication quality parameters of the cells and the train operation status information are collected in time synchronization at a fixed sampling period and the timestamps are aligned. The collected wireless communication quality parameters and train operation status information are arranged in the order of timestamps to form a time-series dataset. Using a fixed time window to segment the time series dataset, the time series data is divided into segments. The data obtained is organized as a vector, and an example of a vector is shown below:

[0010] Where formula (1) is in the middle, and Represents the train's schedule Received from cell The reference signal received power RSRP and the reference signal received quality RSRQ, Representative moment Physical community identifiers Represents the absolute position of the train; Build length is Time series data Used for subsequent predictions, represented as:

[0011] All time-series data are arranged in order of timestamps to form a time-series dataset.

[0012] Preferably, the step involves training a wireless communication quality parameter prediction model using the time-series dataset to obtain a trained multivariate time-series prediction model, and then using this model to generate future... Time series data at time steps include: The time series dataset contains data of length [length]. Time series data The data is input into the wireless communication quality parameter prediction model, which then processes the time-series data. Channel-independent processing is performed, treating the complete temporal sequence of each variable as an independent term. Nonlinear mapping is then applied to each variable sequence using a multilayer perceptron network. The embedded high-dimensional variable terms are then input into a multi-head self-attention module and a feedforward network module, as shown below:

[0013]

[0014]

[0015]

[0016] In formula (3), This represents high-dimensional variable words after independent channel processing and mapping with a multilayer perceptron. Nonlinear mappings representing multilayer perceptron networks; In formula (4), These are query, index, and content matrix, respectively. For the first after feature embedding Each variable represents, For a trainable parameter matrix, The dimension of each attention head; In formula (5), Represents the output of a single attention head. The function calculates the normalized attention weights between the query and the index and applies them to the content matrix; In formula (6), Feature fusion for multi-head attention The results of each attention head are then concatenated. Perform result fusion for learnable parameter matrices; The final output of the encoder architecture is passed through a linear mapping layer to generate predicted wireless communication quality parameters for future time windows. Mean squared error is used as the loss function, and the wireless communication quality parameter prediction model is trained and optimized by minimizing the error between the predicted and true values, as shown below:

[0017] Formula (7) is one of them. Represents the mean square error. for Dimensions These are the actual values ​​and the predicted values, respectively. After training, a well-trained wireless communication quality parameter prediction model is obtained. The trained multivariate time-series prediction model treats the complete time sequence of each variable as an independent term to predict future events. Time series data of length at specific points in time.

[0018] Preferably, the method involves stitching together historical wireless communication quality parameters from urban rail transit scenarios to obtain a linear extrapolation of historical observations, and then using a confidence-weighted mechanism to combine the linear extrapolation of the historical observations with the subsequent future data. The time series data of a certain length are convexly combined to generate state data for future time windows, including: Historical wireless communication quality parameters from urban rail transit scenarios are stitched together, and future parameters are generated using a linear extrapolation method. The linear extrapolation of historical observations over a time step is used to calculate the single-step prediction error of the wireless communication quality parameter prediction model at the current moment, expressed as:

[0019] In formula (8), Represents the single-step prediction error. and represent The predicted value and the actual observed value at time; An exponentially weighted moving average mechanism is used, utilizing the single-step prediction error. The historical smoothed average error is updated to obtain the current smoothed average error, expressed as:

[0020] In formula (9), As a smoothing factor, Represents the historical smoothed average error; Based on the historical smoothed average error Dynamic fusion weights are obtained by monotonically increasing the mapping between preset minimum and maximum weights. These dynamic fusion weights are then used to evaluate the subsequent future... The time series data of the specified length is convexly combined with the linear extrapolation of the historical observations to obtain the fused state data of the future time window, represented as follows:

[0021]

[0022] In formulas (10) and (11), Represents the cropping operation. This represents the preset minimum and maximum weights. Represents the subsequent future output of a multivariate time series prediction model Time series data of length at specific points in time. Linear extrapolation data representing historical observations, This represents the state data for the future time window after merging.

[0023] Preferably, the construction of the actor-critic framework, which includes a policy network and a value network, involves iteratively training a deep reinforcement learning decision model using historically collected experience training data. A proximal policy optimization algorithm is employed to update the parameters of the policy network and value network by minimizing the composite loss function until training converges, resulting in a trained deep reinforcement learning decision model. This includes: An actor-critic framework is constructed, comprising a policy network and a value network. The policy network outputs the action probability distribution, and the value network estimates the long-term expected reward of each state. A deep reinforcement learning decision model is iteratively trained using this actor-critic framework based on historically collected empirical training data. The augmented state space in the policy network consists of historical data and state data from a fused future time window. The assembly is represented as follows:

[0024] In formula (12), Represents historical data, including the current serving cell and the best neighboring cell, both currently and historically. Wireless communication quality parameters for time steps The representative feature mapping function uses a standardization method to perform dimension-by-dimensional normalization on the concatenated multidimensional features; The action space in the policy network is a two-dimensional discrete space, represented as follows:

[0025] In formula (13), This indicates an action to maintain the connection with the current serving cell. This indicates that the action of switching to a nearby cell has been performed; Design a multi-objective reward function and introduce a reward backtracking mechanism. Build as an instant reward With extra rewards The sum of is expressed as:

[0026] The immediate reward includes: a handover penalty, a signal-to-noise ratio reward, and a delay handover penalty, expressed as follows:

[0027] In formula (15), This represents a fixed penalty when performing a switching action. The signal-to-noise ratio bonus is proportional to the signal-to-noise ratio of the current serving cell; This represents a handover delay penalty, which is triggered when the serving cell's RSRP is below a threshold and significantly worse than that of neighboring cells.

[0028]

[0029] In formulas (16) and (17), , and The corresponding reward coefficient in formula (15) The signal-to-interference-to-noise ratio (SIR) represents the signal-to-interference-to-noise ratio of the service cell. and RSRP representing neighboring cells and serving cells, This represents setting a threshold. The additional rewards This includes specific penalties for switching between ping-pong and other electronic games. and the reward items corresponding to the reward backtracking mechanism. , represented as:

[0030]

[0031]

[0032] In formulas (18) to (20), Represents the penalty coefficient for switching between ping-pong games. Weighted allocation of attenuation factor to previous A historical decision-making step, This represents the basic reward for the current step. This represents the number of backtracking steps.

[0033] In the empirical data collected in each iteration, the advantage function is calculated using a generalized advantage estimation mechanism, expressed as:

[0034] In formula (21), Represents timing difference error. Represents the discount factor. The parameter represents the trade-off between deviation and variance; The proximal policy optimization algorithm is used to update the parameters of the policy network and the value network by minimizing the composite loss function, as follows:

[0035] In formula (22), The pruning agent loss represents the policy network. The loss represents the value network. Represents the policy entropy regularization term. , These are the preset weighting coefficients for the loss term; The pruning agent loss limits the magnitude of policy updates by calculating the probability ratio of the old and new policies, and is expressed as:

[0036]

[0037] In formula (23), This represents the clipping range hyperparameter; After multiple mini-batch gradient updates, the parameters of the old policy network are updated to the parameters of the current policy network, and the empirical data is discarded. The next iteration begins, and the training continues until convergence, resulting in a well-trained deep reinforcement learning decision model.

[0038] Preferably, the method of controlling the train to perform cross-zone handover actions during actual operation using a trained deep reinforcement learning decision model includes: using the current time and historical data... Based on the communication parameters and train position of the step, combined with confidence-weighted fusion of future... An enhanced state vector is constructed from the predicted data. This enhanced state vector is then input into a trained deep reinforcement learning decision model. Through real-time reasoning via the deep reinforcement learning decision network, the agent outputs discrete control actions, which the train executes to achieve train zone switching.

[0039] As can be seen from the technical solutions provided by the embodiments of the present invention described above, the present invention improves the stability of the train system, ensures that the train communication link remains at a stable level, and guarantees the quality of communication services under high-speed train movement. The present invention overcomes the short-sighted decision-making and policy oscillation problems caused by the limitation of state information in traditional passive methods, enabling the switching strategy to shift from optimizing instantaneous signals to maintaining long-term train link stability.

[0040] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and will become apparent from the description or may be learned by practice of the invention. Attached Figure Description

[0041] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0042] Figure 1 A flowchart illustrating a train handover control method based on prediction enhancement provided in an embodiment of the present invention; Figure 2 This is a method overview diagram of a train cross-zone handover control method based on prediction enhancement provided in an embodiment of the present invention; Figure 3 A step diagram of the decision algorithm for a train handover control method based on prediction enhancement provided in an embodiment of the present invention. Detailed Implementation

[0043] Embodiments of the present invention are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0044] Those skilled in the art will understand that, unless specifically stated otherwise, the singular forms “a,” “an,” “the,” and “the” used herein may also include the plural forms. It should be further understood that the term “comprising” as used in this specification means the presence of the stated features, integers, steps, operations, elements, and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof. It should be understood that when we say an element is “connected” or “coupled” to another element, it can be directly connected or coupled to the other element, or there may be intermediate elements. Furthermore, “connected” or “coupled” as used herein can include wireless connections or couplings. The term “and / or” as used herein includes any and all combinations of one or more of the associated listed items.

[0045] It will be understood by those skilled in the art that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0046] To facilitate understanding of the embodiments of the present invention, the following will provide further explanation and description with reference to the accompanying drawings and several specific embodiments. These embodiments do not constitute a limitation on the embodiments of the present invention.

[0047] The processing flow of a train handover control method based on prediction enhancement provided in this embodiment of the invention is as follows: Figure 1 As shown, the processing steps include the following: Step S1: Based on the characteristics of train operation in urban rail transit scenarios, periodically collect wireless communication quality parameters and train operation status information of multiple cells during train travel, and use the collected data to form a time-series dataset. Step S2: Use the constructed time series dataset to train the wireless communication quality parameter prediction model to obtain the trained multivariate time series prediction model.

[0048] The wireless communication quality parameter prediction model employs channel-independent processing, treating the complete time-series sequence of each variable as an independent term. Subsequently, a multilayer perceptron network is used for nonlinear mapping, and the embedded high-dimensional variable terms are input into a multi-head self-attention module and a feedforward network module to predict future trends. The time series data represents a specific length of time points. This time series data serves as the state data for generating future time windows and is used as part of the input to the decision model.

[0049] Step S3: The historical wireless communication quality parameters in the urban rail transit scenario are stitched together to obtain a linear extrapolation of the historical observations. A confidence-weighted mechanism is then used to combine the linear extrapolation of the historical observations with the subsequent future predictions from the wireless communication quality parameter prediction model. The time series data of a certain length are convexly combined to generate state data for future time windows. This state data Including the serving cell and the best neighboring cell in the future RSRP and RSRQ values ​​at each time point.

[0050] The state data of the aforementioned future time windows constitute the augmented state space of the deep reinforcement learning decision model.

[0051] Step S4: Construct the augmented state space and action space of the deep reinforcement learning decision model. The augmented state space contains the state data of the future time window mentioned above, and the action space contains the switching action of controlling the train. Design a multi-objective reward function to evaluate the performance of the switching action, and introduce a reward backtracking mechanism to allocate the reward of the current step to the historical decision steps.

[0052] Step S5: Construct an actor-critic framework that includes a policy network and a value network. The policy network is responsible for outputting the action probability distribution, and the value network is responsible for estimating the long-term expected reward of the state. Based on historically collected empirical training data, iteratively train the deep reinforcement learning decision model using the above actor-critic framework. Employ the proximal policy optimization algorithm to update the parameters of the policy network and the value network by minimizing the composite loss function until training converges, thus obtaining the trained deep reinforcement learning decision model.

[0053] The strategy involves using a trained deep reinforcement learning decision model to control the train to perform cross-zone handover actions at appropriate times, and executing cross-zone handover actions during actual operation.

[0054] Preferably, step S1 specifically includes: (1) Data collection scenario: In a fixed scenario where multiple base stations are linearly deployed along the track, the wireless communication quality parameters and train operation status information of multiple cells during the train's movement are periodically collected; (2) Acquisition strategy: The wireless communication quality parameters of the cell and the train operation status information are collected in time synchronization and timestamp aligned at a fixed sampling period; preferably, the sampling granularity is at the millisecond level to match the high-frequency sampling characteristics of vehicle-to-ground wireless communication. (3) Data Organization: The collected wireless communication quality parameters and train operation status information are arranged in timestamp order to form a time-series dataset. The time-series dataset is then segmented using fixed time windows to divide the data into time series sequences. The data obtained is organized as a vector, and an example of a vector is shown below:

[0055] Where formula (1) is in the middle, and Represents the train's schedule Received from cell The reference signal received power (RSRP) and reference signal received quality (RSRQ). Representative moment Physical Cell Identifier (PCI) Represents the absolute position of the train; Therefore, the construction length is... Time series data Used for subsequent predictions, represented as:

[0056] All time-series data are arranged in order of timestamps to form a time-series dataset.

[0057] Preferably, step S2 uses the constructed time-series dataset to train a wireless communication quality parameter prediction model, specifically including: The length is Time series data Input into a wireless communication quality parameter prediction model to predict future trends. Time series data of length at specific time points; Wireless communication quality parameter prediction models for time series data Channel-independent processing is performed, treating the complete temporal sequence of each variable as an independent term. Subsequently, each variable sequence is nonlinearly mapped using a multilayer perceptron network. The embedded high-dimensional variable terms are then input into a multi-head self-attention module and a feedforward network module, represented as follows:

[0058]

[0059]

[0060]

[0061] In formula (3), This represents high-dimensional variable words after independent channel processing and mapping with a multilayer perceptron. Nonlinear mappings representing multilayer perceptron networks; In formula (4), These are query, index, and content matrix, respectively. For the first after feature embedding Each variable represents, For a trainable parameter matrix, The dimension of each attention head; In formula (5), Represents the output of a single attention head. The function calculates the normalized attention weights between the query and the index and applies them to the content matrix; In formula (6), Feature fusion for multi-head attention The results of each attention head are then concatenated. The results are fused into a learnable parameter matrix.

[0062] The final output of the encoder architecture is passed through a linear mapping layer to generate predicted wireless communication quality parameters for future time windows. Mean squared error is used as the loss function, and the wireless communication quality parameter prediction model is trained and optimized by minimizing the error between the predicted and true values, as shown below:

[0063] Formula (7) is one of them. Represents the mean square error. for Dimensions These are the actual value and the predicted value, respectively.

[0064] After training, a trained wireless communication quality parameter prediction model is obtained. This trained multivariate temporal prediction model uses the complete temporal sequence of each variable as an independent term to predict future events. Time series data of length at specific points in time.

[0065] Preferably, step S3 utilizes a confidence-weighted mechanism to perform a convex combination of the linear extrapolation of historical observations and the predicted data, specifically including: Based on historical communication quality parameters, future communication is generated using a linear extrapolation method. The extrapolated value of the time step is used to calculate the single-step prediction error of the wireless communication quality parameter prediction model at the current moment, expressed as:

[0066] In formula (8), Represents the single-step prediction error. and represent The predicted value and the actual observed value at time; Using an exponentially weighted moving average mechanism, the historical smoothed average error is updated using the single-step prediction error to obtain the smoothed average error at the current moment, expressed as:

[0067] In formula (9), As a smoothing factor, Represents the historical smoothed average error; Based on the smoothed average error, the dynamic fusion weights are calculated using a monotonically increasing mapping and constrained. Between the preset minimum and maximum weights, the final fusion weight is obtained. This dynamic fusion weight is then used to perform a convex combination of the predicted communication quality parameter and the extrapolated value to obtain the fused future state data, represented as follows:

[0068]

[0069] In formulas (10) and (11), Represents the cropping operation. This represents the preset minimum and maximum weights. and This represents the output data of the predictive model and the linear extrapolated data. This represents the state data for the future time window after merging.

[0070] Preferably, the construction of the enhanced state space and action space in step S4 specifically includes: The enhanced state space is composed of historical data and the fused future state data generated in step S3, and is represented as follows:

[0071] In formula (12), Represents historical data, including the current serving cell and the best neighboring cell, both currently and historically. Wireless communication quality parameters for time steps The representative feature mapping function uses a standardization method to perform dimension-by-dimensional normalization on the concatenated multidimensional features; Construct an action space, which is defined as a two-dimensional discrete space, represented as:

[0072] In formula (13), This indicates an action to maintain the connection with the current serving cell. This indicates that the action of switching to a nearby cell will be performed.

[0073] Preferably, step S5 specifically includes: Step S5 involves designing a multi-objective reward function and introducing a reward backtracking mechanism, as detailed below: The multi-objective reward function Build as an instant reward With extra rewards The sum of is expressed as:

[0074] The immediate reward includes: a handover penalty, a signal-to-noise ratio reward, and a delay handover penalty, expressed as follows:

[0075] In formula (15), This represents a fixed penalty when performing a switching action. The signal-to-noise ratio bonus is proportional to the signal-to-noise ratio of the current serving cell; This represents a handover delay penalty, which is triggered when the serving cell's RSRP is below a threshold and significantly worse than that of neighboring cells.

[0076]

[0077] In formulas (16) and (17), , and The corresponding reward coefficient in formula (15) The signal-to-interference-to-noise ratio (SIR) represents the signal-to-interference-to-noise ratio of the service cell. and RSRP representing neighboring cells and serving cells, This represents setting a threshold. The additional rewards This includes specific penalties for switching between ping-pong and other electronic games. and the reward items corresponding to the reward backtracking mechanism. , represented as:

[0078]

[0079]

[0080] In formulas (18) to (20), Represents the penalty coefficient for switching between ping-pong games. Weighted allocation of attenuation factor to previous A historical decision-making step, This represents the basic reward for the current step. This represents the number of backtracking steps.

[0081] Preferably, step S6 iteratively trains the deep reinforcement learning decision model, as follows: An actor-critic framework is adopted, which includes a policy network and a value network. The policy network is responsible for outputting the action probability distribution, and the value network is responsible for estimating the long-term expected reward of the state. In each iteration, the generalized advantage estimation mechanism is used to calculate the advantage function from the collected empirical data, expressed as:

[0082] In formula (21), Represents timing difference error. Represents the discount factor. The parameter represents the trade-off between deviation and variance; The proximal policy optimization algorithm is used to update the parameters of the policy network and the value network by minimizing the composite loss function, as follows:

[0083] In formula (22), The pruning agent loss represents the policy network. The loss represents the value network. Represents the policy entropy regularization term. , Preset weighting coefficients for loss terms; The pruning agent loss limits the magnitude of policy updates by calculating the probability ratio of the old and new policies, and is expressed as:

[0084]

[0085] In formula (23), This represents the clipping range hyperparameter; After multiple mini-batch gradient updates, the parameters of the old policy network are updated to the parameters of the current policy network, and the empirical data is discarded. The next iteration begins, and the training continues until convergence, resulting in a well-trained deep reinforcement learning decision model.

[0086] In actual operation, the trained deep reinforcement learning decision-making model uses the current moment and historical data... Based on the communication parameters and train position of the step, combined with confidence-weighted fusion of future... An enhanced state vector is constructed from predicted data and used as input. This vector is then inferred in real-time by a deep reinforcement learning decision network, directly outputting discrete control actions. This enables train handover control based on forward-looking information. It allows the train to perform handover actions at appropriate times, improving the quality of the wireless communication link and ensuring continuous and reliable data transmission.

[0087] Example 1

[0088] Step S1: Periodically collect the train's wireless communication quality parameters and train operation status information; Step S2: Use the constructed time series dataset to train a multivariate time series prediction model; Step S3: Using a confidence weighting mechanism, perform a convex combination of the observed linear extrapolation and the predicted data; Step S4: Construct the augmented state space and action space of the deep reinforcement learning decision model; Step S5: Design a multi-objective reward function and introduce a reward backtracking mechanism; Step S6: Iteratively train the final switching control model.

[0089] Figure 2 This diagram provides an overview of a train handover control method based on prediction enhancement, as provided in an embodiment of the present invention. The framework consists of two core components: a time-series prediction model and an enhanced deep reinforcement learning decision model. First, during predictor training, historical data including wireless communication parameters and train operating states is used to train the predictor through the iTransformer module, resulting in a trained prediction model. Second, in the handover decision model, the agent acquires a state from the environment. This state space is composed of historical data and future data generated using the trained predictor and fused with a confidence-weighted fusion mechanism. The agent outputs an action based on this enhanced state, and the environment returns a reward. Through iterative optimization using a deep reinforcement learning algorithm, the final handover control model is obtained.

[0090] Figure 3This is a complete training flowchart of a handover control algorithm provided in an embodiment of the present invention. The process begins by loading a pre-trained wireless communication quality parameter prediction model and initializing the policy network and value network. In the main training loop, it first interacts with the environment: using historical data and future prediction parameters generated by the predictor, it generates fused future state data through a confidence-weighted mechanism. Subsequently, an enhanced state space is constructed, and the policy network samples handover actions accordingly. After the agent executes the action, it receives multi-objective rewards and stores this experience in the data pool. After T steps of interaction, it enters the optimization phase: calculating the advantage function of all experiences and iterating K times, using a deep reinforcement learning algorithm to batch update the parameters of the policy and value networks. After optimization, the old and new policy networks are synchronized, the data pool is cleared, and the process returns to the interaction phase until the maximum number of training iterations is reached, ultimately obtaining a trained handover control model.

[0091] In summary, this invention provides a prediction-enhanced end-to-end handover control framework. This invention combines a multivariate temporal prediction model with a deep reinforcement learning decision-making model. This framework synergistically integrates the forward-looking nature of channel prediction with the dynamic decision-making capabilities of deep reinforcement learning, overcoming the short-sighted decision-making and policy oscillation problems caused by the limitation of state information in traditional passive methods. This allows the handover strategy to shift from optimizing instantaneous signals to maintaining long-term link stability.

[0092] This invention constructs an enhanced state space that integrates future states and introduces a confidence-weighted mechanism. The invention concatenates historical wireless communication quality parameters with the fused future state data to construct the agent's enhanced state space. The fusion process employs a confidence-weighted mechanism, which performs a convex combination of the model's predicted data and the linear extrapolated data from historical observations based on the prediction accuracy of the prediction model within a preset historical time window. This design balances historical real data with future predictions, mitigating the negative impact of inaccurate predictions.

[0093] This invention designs a multi-objective reward function and introduces a reward backtracking mechanism. The multi-objective reward function includes handover penalties, signal-to-noise ratio rewards, delayed handover penalties, and ping-pong handover-specific penalties to comprehensively evaluate the performance of handover actions. Simultaneously, a reward backtracking mechanism is introduced, distributing the reward of the current step to previous historical decision steps in a decaying manner, thus resolving the credit allocation problem caused by reward delays in handover decisions.

[0094] Those skilled in the art will understand that the accompanying drawings are merely schematic diagrams of one embodiment, and the modules or processes shown in the drawings are not necessarily essential for implementing the present invention.

[0095] As can be seen from the above description of the embodiments, those skilled in the art can clearly understand that the present invention can be implemented by means of software plus necessary general-purpose hardware platforms. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in various embodiments or some parts of the embodiments of the present invention.

[0096] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, for apparatus or system embodiments, since they are basically similar to method embodiments, the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. The apparatus and system embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without creative effort.

[0097] The above description is merely a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A train handover control method based on predictive enhancement, characterized in that, include: In response to the characteristics of train operation in urban rail transit scenarios, wireless communication quality parameters and train operation status information of multiple cells are periodically collected during the train's journey, and the collected data are used to construct a time-series dataset. A wireless communication quality parameter prediction model is trained using the aforementioned time series dataset to obtain a trained multivariate time series prediction model. This trained model is then used to generate future... Time series data at time points with time steps; Historical wireless communication quality parameters from urban rail transit scenarios are stitched together to obtain a linear extrapolation of historical observations. A confidence-weighted mechanism is then used to correlate this linear extrapolation with subsequent future data. Convex combination of time series data at time points of length is used to generate state data for future time windows; An actor-critic framework comprising a policy network and a value network is constructed. Based on historically collected experience training data, a deep reinforcement learning decision model is iteratively trained using the actor-critic framework. A proximal policy optimization algorithm is employed to update the parameters of the policy network and the value network by minimizing the composite loss function until training converges, resulting in a well-trained deep reinforcement learning decision model. The well-trained deep reinforcement learning decision model is then used to control the train to perform cross-zone handover actions during actual operation.

2. The method according to claim 1, characterized in that, The aforementioned method, targeting the characteristics of train operation in urban rail transit scenarios, involves periodically collecting wireless communication quality parameters and train operation status information from multiple cells during train travel. The collected data is then used to construct a time-series dataset, including: In a fixed scenario where multiple base stations are linearly deployed along a track, the wireless communication quality parameters of the cells and the train operation status information are collected in time synchronization at a fixed sampling period and the timestamps are aligned. The collected wireless communication quality parameters and train operation status information are arranged in the order of timestamps to form a time-series dataset. Using a fixed time window to segment the time series dataset, the time series data is divided into segments. The data obtained is organized as a vector, and an example of a vector is shown below: Where formula (1) is in the middle, and Represents the train's schedule Received from cell The reference signal received power RSRP and the reference signal received quality RSRQ, Representative moment Physical community identifiers Represents the absolute position of the train; Build length is Time series data Used for subsequent predictions, represented as: All time-series data are arranged in order of timestamps to form a time-series dataset.

3. The method according to claim 2, characterized in that, The process involves training a wireless communication quality parameter prediction model using the time-series dataset to obtain a trained multivariate time-series prediction model, and then using this model to generate future... Time series data at time steps include: The time series dataset contains data of length [length]. Time series data The data is input into the wireless communication quality parameter prediction model, which then processes the time-series data. Channel-independent processing is performed, treating the complete temporal sequence of each variable as an independent term. Nonlinear mapping is then applied to each variable sequence using a multilayer perceptron network. The embedded high-dimensional variable terms are then input into a multi-head self-attention module and a feedforward network module, as shown below: In formula (3), This represents high-dimensional variable words after independent channel processing and mapping with a multilayer perceptron. Nonlinear mappings representing multilayer perceptron networks; In formula (4), These are query, index, and content matrix, respectively. For the first after feature embedding Each variable represents, For a trainable parameter matrix, The dimension of each attention head; In formula (5), Represents the output of a single attention head. The function calculates the normalized attention weights between the query and the index and applies them to the content matrix; In formula (6), Feature fusion for multi-head attention The results of each attention head are then concatenated. Perform result fusion for learnable parameter matrices; The final output of the encoder architecture is passed through a linear mapping layer to generate predicted wireless communication quality parameters for future time windows. Mean squared error is used as the loss function, and the wireless communication quality parameter prediction model is trained and optimized by minimizing the error between the predicted and true values, as shown below: Formula (7) Represents the mean square error. for Dimensions These are the actual values ​​and the predicted values, respectively. After training, a well-trained wireless communication quality parameter prediction model is obtained. The trained multivariate time-series prediction model treats the complete time sequence of each variable as an independent term to predict future events. Time series data of length at specific time points.

4. The method according to claim 3, characterized in that, The method involves stitching together historical wireless communication quality parameters from urban rail transit scenarios to obtain a linear extrapolation of historical observations. A confidence-weighted mechanism is then used to combine this linear extrapolation with subsequent future data. The time series data of a certain length are convexly combined to generate state data for future time windows, including: Historical wireless communication quality parameters from urban rail transit scenarios are stitched together, and future parameters are generated using a linear extrapolation method. The linear extrapolation of historical observations over a time step is used to calculate the single-step prediction error of the wireless communication quality parameter prediction model at the current moment, expressed as: In formula (8), Represents the single-step prediction error. and represent The predicted value and the actual observed value at time; An exponentially weighted moving average mechanism is used, utilizing the single-step prediction error. The historical smoothed average error is updated to obtain the current smoothed average error, expressed as: In formula (9), As a smoothing factor, Represents the historical smoothed average error; Based on the historical smoothed average error Dynamic fusion weights are obtained by monotonically increasing the mapping between preset minimum and maximum weights. These dynamic fusion weights are then used to evaluate the subsequent future... The time series data of the specified length is convexly combined with the linear extrapolation of the historical observations to obtain the fused state data of the future time window, represented as follows: In formulas (10) and (11), Represents the cropping operation. This represents the preset minimum and maximum weights. Represents the subsequent future output of a multivariate time series prediction model Time series data of length at specific points in time. Linear extrapolation data representing historical observations, This represents the state data for the future time window after merging.

5. The method according to claim 4, characterized in that, The aforementioned construction includes an actor-critic framework comprising a policy network and a value network. Based on historically collected empirical training data, the deep reinforcement learning decision model is iteratively trained using this actor-critic framework. A proximal policy optimization algorithm is employed to update the parameters of the policy network and value network by minimizing the composite loss function until training converges, resulting in a well-trained deep reinforcement learning decision model. This includes: An actor-critic framework is constructed, comprising a policy network and a value network. The policy network outputs the action probability distribution, and the value network estimates the long-term expected reward of each state. A deep reinforcement learning decision model is iteratively trained using this actor-critic framework based on historically collected empirical training data. The augmented state space in the policy network consists of historical data and state data from a fused future time window. The assembly is represented as follows: In formula (12), Represents historical data, including the current serving cell and the best neighboring cell, both currently and historically. Wireless communication quality parameters for time steps The representative feature mapping function uses a standardization method to perform dimension-by-dimensional normalization on the concatenated multidimensional features; The action space in the policy network is a two-dimensional discrete space, represented as follows: In formula (13), This indicates an action to maintain the connection with the current serving cell. This indicates that the action of switching to a nearby cell has been performed; Design a multi-objective reward function and introduce a reward backtracking mechanism. Build as an instant reward With extra rewards The sum of is expressed as: The immediate reward includes: a handover penalty, a signal-to-noise ratio reward, and a delay handover penalty, expressed as follows: In formula (15), This represents a fixed penalty when performing a switching action. The signal-to-noise ratio bonus is proportional to the signal-to-noise ratio of the current serving cell; This represents a handover delay penalty, which is triggered when the serving cell's RSRP is below a threshold and significantly worse than that of neighboring cells. In formulas (16) and (17), , and The corresponding reward coefficient in formula (15) The signal-to-interference-to-noise ratio (SIR) represents the signal-to-interference-to-noise ratio of the service cell. and RSRP representing neighboring cells and serving cells, This represents setting a threshold. The additional rewards This includes specific penalties for switching between ping-pong and other electronic games. and the reward items corresponding to the reward backtracking mechanism. , represented as: In formulas (18) to (20), Represents the penalty coefficient for switching between ping-pong games. Weighted allocation of attenuation factor to previous A historical decision-making step, This represents the basic reward for the current step. Represents the number of backtracking steps; In the empirical data collected in each iteration, the advantage function is calculated using a generalized advantage estimation mechanism, expressed as: In formula (21), Represents timing difference error. Represents the discount factor. The parameter represents the trade-off between deviation and variance; The proximal policy optimization algorithm is used to update the parameters of the policy network and the value network by minimizing the composite loss function, as follows: In formula (22), The pruning agent loss represents the policy network. The loss represents the value network. Represents the policy entropy regularization term. , These are the preset weighting coefficients for the loss term; The pruning agent loss limits the magnitude of policy updates by calculating the probability ratio of the old and new policies, and is expressed as: In formula (23), This represents the clipping range hyperparameter; After multiple mini-batch gradient updates, the parameters of the old policy network are updated to the parameters of the current policy network, and the empirical data is discarded. The next iteration begins, and the training continues until convergence, resulting in a well-trained deep reinforcement learning decision model.

6. The method according to claim 5, characterized in that, The method of using a trained deep reinforcement learning decision model to control the train to perform cross-zone handover actions during actual operation includes: In the present moment and history Based on the communication parameters and train position of the step, combined with confidence-weighted fusion of future... An enhanced state vector is constructed from the predicted data. This enhanced state vector is then input into a trained deep reinforcement learning decision model. Through real-time reasoning via the deep reinforcement learning decision network, the agent outputs discrete control actions, which the train executes to achieve train zone switching.

Images