A formation collaborative tracking control method, device, equipment, medium and product

Abstract

The present application discloses a formation cooperative tracking control method, device, equipment, medium and product, which relates to the field of tracking control. The method includes: obtaining the operation data of a virtual marshaling train; inputting the operation data into a deep neural network dynamics model and updating the dynamics model parameters; using a reinforcement learning method, according to the train operation data and the updated dynamics model, the elastic optimal control objective function of the virtual marshaling train is determined based on the elastic interval evaluation model; the elastic interval evaluation model is a mathematical model for cooperative operation based on the calculation of the safe following distance based on the relative braking distance between two adjacent trains to set the interval following distance; the cross entropy algorithm is used to solve the elastic optimal control objective function to obtain a control sequence to perform formation cooperative tracking control on the virtual marshaling train. The present application aims to improve the accuracy and stability of formation cooperative tracking control.

Description

Technical Field

[0001] The present application relates to the field of tracking control, and in particular to a formation collaborative tracking control method, device, equipment, medium and product. Background Art

[0002] Virtual train formation is a novel rail transport organization method that uses communication and control technologies to virtually combine multiple trains for operation, improving transport efficiency, reducing energy consumption, and lowering operating costs. However, in actual operation, trains are subject to a variety of uncertain interference factors, such as track irregularities, wind, vehicle failures, and communication delays. These interference factors can cause trains to deviate from their intended trajectory, thereby affecting the coordinated tracking performance of the train formation.

[0003] Due to the existence of various interference factors during train operation, it is crucial to improve the accuracy and stability of the collaborative tracking control of virtual trains considering uncertain interference factors. Summary of the Invention

[0004] The purpose of this application is to provide a formation collaborative tracking control method, device, equipment, medium and product, which can improve the accuracy and stability of formation collaborative tracking control.

[0005] To achieve the above objectives, this application provides the following solutions:

[0006] In a first aspect, the present application provides a formation cooperative tracking control method, comprising:

[0007] Acquiring operation data of a virtual marshaled train; the virtual marshaled train is a coupled train cluster composed of multiple single-point trains;

[0008] Inputting the operating data into a deep neural network dynamics model to obtain output state data; the deep neural network dynamics model is obtained by training the deep neural network using historical operating data and corresponding output state data as an offline data set;

[0009] A reinforcement learning method is used to determine the elastic optimal control objective function of the virtual marshaling train based on the output state data and a flexible interval evaluation model; the flexible interval evaluation model is a mathematical model for cooperative operation based on calculating a safe following distance based on the relative braking distance between two adjacent trains to set a following distance;

[0010] The cross entropy algorithm is used to solve the elastic optimal control objective function to obtain a control sequence; the control sequence is used to perform formation cooperative tracking control on the virtual marshaled train.

[0011] In a second aspect, the present application provides a formation cooperative tracking control device, comprising:

[0012] A data acquisition module is used to acquire the operation data of a virtual marshaling train; the virtual marshaling train is a coupled train cluster composed of multiple single-point trains;

[0013] An output module is used to input the operating data into a deep neural network dynamics model to obtain output state data; the deep neural network dynamics model is obtained by training the deep neural network using historical operating data and corresponding output state data as an offline data set;

[0014] an objective function determination module for determining, using a reinforcement learning method, a flexible optimal control objective function for a virtual marshaled train based on the output state data and a flexible interval evaluation model; the flexible interval evaluation model is a mathematical model for cooperative operation based on a set interval and following distance and calculating a safe following distance based on a relative braking distance between two adjacent trains;

[0015] A solution module is used to solve the elastic optimal control objective function using a cross entropy algorithm to obtain a control sequence; the control sequence is used to perform formation collaborative tracking control on the virtual marshaled train.

[0016] In a third aspect, the present application provides a computer device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the above-described formation collaborative tracking control method.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-mentioned formation collaborative tracking control method.

[0018] In a fifth aspect, the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned formation collaborative tracking control method.

[0019] According to the specific embodiments provided in this application, this application discloses the following technical effects:

[0020] The present application provides a formation cooperative tracking control method, apparatus, device, medium, and product. The method involves obtaining operational data of a virtual train; inputting the operational data into a deep neural network dynamics model to obtain output state data; and employing a reinforcement learning method to determine the elastic optimal control objective function of the virtual train based on the output state data and a flexible interval evaluation model. The deep neural network dynamics model and the reinforcement learning method ensure that the virtual train achieves the desired formation operational state and prevents collisions between adjacent trains, thereby ensuring safe train operation. The flexible interval evaluation model is a mathematical model for cooperative operation based on the calculated safe following distance based on the relative braking distance between two adjacent trains, thereby setting the interval following distance. The flexible interval evaluation model ensures dynamic tracking distance adjustment between adjacent trains. Due to the nonlinear characteristics of train dynamics and reward functions, which can result in low accuracy, the present application employs a cross-entropy algorithm to solve the elastic optimal control objective function, improving the accuracy of formation cooperative tracking control and obtaining a control sequence, thereby performing formation cooperative tracking control on the virtual train. Consequently, the present application improves the accuracy and stability of formation cooperative tracking control. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0022] Figure 1 This is a flow chart of the formation cooperative tracking control method;

[0023] Figure 2 Schematic diagram of the tracking interval mechanism of virtual marshaling trains based on relative braking distance;

[0024] Figure 3 It is a schematic diagram of the dynamic grouping process;

[0025] Figure 4 It is a schematic diagram of the collaborative operation process;

[0026] Figure 5 It is a schematic diagram of the dynamic decompilation process;

[0027] Figure 6 A block diagram of the predictive control architecture for deep reinforcement learning data-driven models;

[0028] Figure 7 It is the DNN structure diagram;

[0029] Figure 8 Schematic diagram of RL control process;

[0030] Figure 9 It is the historical running data curve of the train;

[0031] Figure 10 This is a schematic diagram of speed prediction results;

[0032] Figure 11 Schematic diagram of speed prediction error;

[0033] Figure 12 Schematic diagram of displacement prediction results;

[0034] Figure 13 Schematic diagram of displacement prediction error;

[0035] Figure 14 This is the leader-follower train speed curve in scenario 1;

[0036] Figure 15 This is the leader-follower train spacing curve for scenario 1;

[0037] Figure 16 is the leader-follower train state tracking error curve in scenario 1;

[0038] Figure 17 This is the leader-follower train acceleration curve in scenario 1;

[0039] Figure 18 This is the leader-follower train speed curve in scenario 2;

[0040] Figure 19 This is the leader-follower train spacing curve for scenario 2;

[0041] Figure 20 is the leader-follower train state tracking error curve in scenario 2;

[0042] Figure 21 Figure 2 shows the acceleration curve of the leader-follower train in scenario 2. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0044] In order to make the above-mentioned purposes, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below with reference to the accompanying drawings and specific implementation methods.

[0045] In an exemplary embodiment, Figure 1 As shown, a formation cooperative tracking control method is provided, and the formation cooperative tracking control method includes:

[0046] Step 100: Acquire the operation data of a virtual train set. The virtual train set is a coupled train cluster composed of multiple single-point trains.

[0047] Step 200: Input the operating data into the deep neural network dynamics model to obtain output state data. The deep neural network dynamics model is obtained by training the deep neural network using historical operating data and corresponding output state data as offline data sets.

[0048] Step 300: Using reinforcement learning, determine the flexible optimal control objective function for the virtual train formation based on the output state data and the flexible interval evaluation model. The flexible interval evaluation model is a mathematical model for cooperative operation based on the calculated safe following distance based on the relative braking distance between two adjacent trains.

[0049] Elastic optimal control objective function, specifically including:

[0050] .

[0051] in, is the elastic optimal control objective function; is the prediction time domain; is the sampling time; To predict any moment in the time domain; for Time Train The reward function of for Time Train The operating status of for Time Train control input.

[0052] Step 400: Using a cross entropy algorithm to solve the elastic optimal control objective function, a control sequence is obtained. The control sequence is used to perform formation cooperative tracking control on the virtual marshaled train.

[0053] The method for determining the deep neural network dynamics model specifically includes:

[0054] Obtain an offline data set; normalize the offline data set to obtain a processed data set; construct a deep neural network; input the processed data set into the deep neural network, use the ADAM algorithm to minimize the loss function, train the parameters of the deep neural network, and obtain a trained deep neural network; the parameters include: weights.

[0055] The trained deep neural network is determined as a deep neural network dynamics model.

[0056] In one embodiment, a minimum-maximum normalization method is used to normalize the offline data set to obtain a processed data set; the expression of the processed data set is:

[0057] .

[0058] in, is the processed data set; It is an offline dataset; is the minimum value in the offline dataset; and These are all sample numbers in the offline dataset; is the total number of samples in the offline dataset; is the maximum value in the offline dataset.

[0059] The expression of the loss function is:

[0060] .

[0061] in, is the loss function; is the sampling time; is the sampling time; For trains In the The operating status at the sampling time; For trains In the The operating status at the sampling time; For trains In the Control input at the sampling time; For learning models; 、 are all weight coefficients of deep neural networks.

[0062] In one embodiment, a cross entropy algorithm is used to solve the elastic optimal control objective function to obtain a control sequence, which specifically includes:

[0063] Determine the initial parameters; the initial parameters include: setting the iteration time, ratio, and N train samples; the train samples are collected through parameterized distribution based on the probability distribution function.

[0064] The fitness function is determined according to the elastic optimal control objective function.

[0065] The fitness function value is determined based on the fitness function according to the parameter combination at the current iteration time; the parameter combination includes: ratio and train sample.

[0066] Determine whether the iteration stopping condition is met; the iteration stopping condition is that the current iteration time reaches the set iteration time or the fitness function value at the current iteration time is within the set elite threshold range.

[0067] If not, the elite sample set is determined based on the fitness function value, and the parameters of the probability distribution function are updated based on the set learning step parameters, and "the fitness function value is determined based on the fitness function according to the parameter combination at the current iteration time".

[0068] If so, the control sequence is determined according to the parameter combination at the current iteration time.

[0069] In practical applications, the operating steps corresponding to the solution mentioned in this application are as follows:

[0070] Firstly, a flexible interval evaluation model for virtual marshaling trains was established. Through this model, it was determined whether adjacent trains were in a coupled operation state, and the train operation state was adjusted online.

[0071] Although the control method designed in this application does not require a mathematical model of a virtual train, in order to better understand its operating principle, a brief introduction to the train dynamics is given here. A virtual train is described as a coupled train cluster consisting of multiple single-point trains, i.e., the forces between carriages in a unit train are ignored. According to Newton's second law, the unit train dynamics model can be described as:

[0072] (1)

[0073] in, and Trains Position and velocity at the current moment; and Trains The mass and the mass coefficient of gyration; for The first derivative of ; for The first derivative of . For the serial number.

[0074] For trains exist The input control force (including traction and braking force) received at all times; It is the basic resistance encountered by the train, mainly caused by the running resistance of the train's internal components, air resistance, friction between the wheels and rails, etc. Its characteristics are related to the train speed, and It can be approximately expressed as:

[0075] (2)

[0076] in, 、 、 These are the basic operating resistance coefficients, which can be obtained based on past experience; The additional resistance to the train is closely related to the environment in which the train runs. It usually includes three parts: slope additional resistance, curve additional resistance and tunnel additional resistance. Expressed as:

[0077] (3)

[0078] in, is the gravitational acceleration of the train; 、 、 are track slope, track curve radius and tunnel length respectively. is an empirical constant.

[0079] Flexible interval evaluation model: In the virtual marshaling operation mode, each unit train runs in a coordinated manner with a small interval following distance. The optimal safe following distance is solved by calculating the relative braking distance between two adjacent trains in real time, such as Figure 2 As shown, the actual running distance of trains and minimum safety distance The calculation formula is as follows:

[0080] (4)

[0081] in, is the body length of the preceding train; for The common braking distance of the train following the time is expressed as ; for The emergency braking distance of the train ahead at any moment is usually expressed as ; for The distance travelled by the train during the instantaneous reaction time; This is the parking safety margin. For trains displacement; For trains Common braking control acceleration; For trains Common braking controls acceleration. For trains exist The speed of time.

[0082] Further design of optimal tracking spacing for:

[0083] (5)

[0084] in, is the optimal spacing adjustment coefficient. Then the elasticity between adjacent trains Designed to:

[0085] (6)

[0086] When the train elasticity is small, the actual distance between the two trains is large, and the trains may be in an uncoupled state; when the train elasticity approaches "1", the actual distance between the two trains gradually approaches the optimal distance, and the trains meet one of the conditions for entering the coupled state; when the train elasticity is large, the actual distance between the two trains is small, and the trains are in a dangerous state, which may lead to safety accidents such as collisions.

[0087] "Three-stage" marshaling process: Virtual marshaling consists of three stages: dynamic marshaling, coordinated operation, and dynamic unmarshaling. To ensure punctuality and safety, both dynamic marshaling and unmarshaling must be completed within designated sections.

[0088] Dynamic marshaling means that the train switches from independent operation state to coupled operation state. Taking the two-train marshaling as an example, Figure 3 As shown, "LT" indicates the leader train (front train); "FT" indicates the follower train (back train); Indicates the expected spacing between two trains in virtual marshaling mode; Indicates the expected distance during the independent operation phase. During dynamic marshaling, the train needs to reduce the tracking distance with the preceding train to achieve train-to-train communication. When the communication range is met, the train needs to maintain the same speed as the preceding train. There are two dynamic marshaling methods: In the first case, it is assumed that both trains are running at the maximum speed at the beginning. It runs independently on the line. When the rear train completely enters the marshaling section, it will send a marshaling request signal to the front train. After the front train receives the signal, it will reduce its speed to a certain value so that the rear train can catch up with the front train. When the distance between the two trains is about to reach the expected distance in the coordinated operation stage, the front train starts to accelerate to Keep the same speed as the car behind. In the second case, suppose both cars are running at the same speed at the beginning. It runs independently on the line. When the rear car completely enters the marshaling section, it will also send a marshaling request signal to the front car to let the front car prepare for marshaling. At this time, the rear car will increase its speed to , in order to catch up with the front train, when the distance between the two trains is about to reach the expected distance in the coordinated operation phase, the rear train starts to slow down to Keep the same speed as the preceding train. After the dynamic marshaling is completed, the train group immediately enters the coordinated operation stage. In this stage, the train control system needs to achieve the goal of speed coordination of adjacent trains and ensure the desired running distance. This process is as follows Figure 4 When the virtual marshaling train enters the dynamic disassembly section, the running distance between the disassembly train and the preceding train needs to be increased until the required distance for the independent running state mode is reached, as shown in Figure 5 As shown, this process is actually the inverse process of dynamic grouping. Its unmarshalling process is similar to the dynamic grouping process, so it will not be described in detail here.

[0089] Secondly, a more accurate data-driven deep neural network (DNN) was constructed. By using the recorded train operation input and output data, a train dynamics model was trained. A reinforcement learning distributed predictive control method was designed. This control method introduced the concept of reinforcement learning on the basis of distributed model predictive control. Combined with the elastic interval evaluation model, the elastic optimal control objective function of the virtual marshaling train was designed. The optimal control problem of the train was constructed based on the trained DNN and the objective function, and the cross-entropy method (CEM) algorithm was used to solve the optimal control problem.

[0090] like Figure 6 The following is a block diagram of the deep reinforcement learning control structure based on the data-driven model designed in this application. Offline learning of train models, and in actual operation, through the collected real-time data The train model is updated online to realize the construction of an "offline + online" data-driven precision model. The reinforcement learning reward function is introduced into the performance indicator function of the MPC controller, and the tracking spacing of the virtual train is flexibly adjusted, thereby realizing deep reinforcement learning data-driven model predictive control.

[0091] Dynamical models of deep neural networks.

[0092] For a virtual train, its unit train state vector Can be defined as , control input Defined as , the train dynamics model is expressed as follows:

[0093] (9)

[0094] in, For trains i Derivatives of state quantities; For The relevant functions are specifically expressed as .

[0095] Since the data-driven model designed in this application is trained based on discrete sampling data, the discretized train dynamics model is expressed as:

[0096] (10)

[0097] in, For trains In the The operating status at the sampling time; For trains In the The operating status at the sampling time; For trains In the Control input at the sampling time; is the state transition function; is the sampling time interval.

[0098] According to the current operation status information of the high-speed train , a deep neural network is used to learn the state transfer function of the train, and the learning model can be expressed as ,in, is the weight coefficient of the DNN model, and the model output is the predicted state deviation at the next moment Therefore, the prediction model It can be expressed as:

[0099] (11)

[0100] DNN usually consists of three parts: input layer, hidden layer and output layer. Figure 7 As shown, DNN converts the information received by the input layer into and weight coefficient 、 Pass forward together, and then combine the activation function The results are distributed to each neuron in the hidden layer and passed forward in sequence to the output layer. The network uses historical train operation data to approximate the actual train model, thereby accurately predicting the train's operating status at the next moment.

[0101] The sampled historical train operation data is used as an offline dataset for training DNN as follows:

[0102] (12)

[0103] in, For trains The sample data set of For trains The initial state data set includes the train's initial speed information and initial position information at each sampling moment, that is, the train's state information at the previous moment; For trains The control input data set includes the train control input information at each sampling moment; For trains The output state data set includes the train final speed information and final position information at each sampling moment, that is, the actual state information of the train at the sampling moment. To ensure the accuracy of the model, a new data set is collected in real time during the train operation to train the DNN online. Therefore, the DNN training data set is composed of the offline data set. With online datasets It consists of two parts.

[0104] As mentioned above, the input of DNN includes speed (unit: ),Location (unit: ) and control acceleration (unit: Since these three types of input data are in different scales, this may lead to low efficiency in training the DNN model. Therefore, in order to speed up the model training, the input data needs to be normalized. This application chooses the minimum-maximum normalization method, and its mathematical expression is described as follows:

[0105] (13)

[0106] The normalized data is input into the DNN for training. During the DNN training process, this application uses the ADAM algorithm to minimize the loss function:

[0107] (14)

[0108] Therefore, the training goal of the DNN model is to find the optimal weight coefficient Make the loss function minimize.

[0109] Distributed predictive control with reinforcement learning.

[0110] In order to realize the collaborative formation control of virtual marshaled trains, this application proposes a distributed model predictive control algorithm based on reinforcement learning. This method can ensure that the virtual marshaled trains reach the desired formation operation state and prevent collisions between adjacent trains to ensure the safe operation of the trains.

[0111] The control objective of this application is to achieve coordinated control of dynamic interval adjustment of virtual trains. That is, the controller must not only ensure that the operating status of each unit train converges and stabilizes within a certain period of time, but also ensure dynamic tracking distance adjustment between adjacent trains to avoid safety accidents in response to uncertain interference and other issues during operation. At the same time, the controller output acceleration must meet the requirements of the high-speed train operation control system. Therefore, the control objective can be specifically described as:

[0112] (15)

[0113] in, is the maximum braking deceleration of the train; For trains The acceleration at the current moment; is the maximum traction acceleration of the train. For trains The speed at the current moment.

[0114] Reinforcement learning (RL) is a machine learning method for sequential decision making. Figure 8 As shown in the figure, in the reinforcement learning control process, the agent and the environment are always in an interactive state. The agent obtains state information from the environment and uses this state information to act on the environment. Then the environment outputs the next state and the corresponding action reward based on the action. The agent, or decision maker, is usually used to explore appropriate actions to maximize the total reward obtained from the environment. In this application, the controller represents the agent and the train represents the environment. In the actual operation control process, it is assumed that is the state vector of the train at the current moment, is the reward value of the train at the current moment, and the controller obtains a control action based on these two values Act on the train to get the state vector of the train at the next moment And the reward function , and repeat the process.

[0115] The reward function constructed in this application It includes two parts: collaboration reward and collision avoidance penalty.

[0116] First, according to formula (15), in order to maintain a stable train formation, three reward functions need to be set, including consistent speed reward, flexible interval reward, and control variable reward, which are described as follows:

[0117] (16)

[0118] (17)

[0119] (18)

[0120] in, 、 、 They are consistent speed reward, flexible interval reward and control variable reward; For the following train With the moving train Speed ​​deviation; For trains Deviation of elasticity from the optimal elasticity; 、 、 、 They are the weight coefficients of the corresponding reward function, and the importance of speed, spacing, control amount and control change is reflected by designing the sizes of the four. For trains i exist t Flexibility at all times.

[0121] Secondly, in order to avoid collisions between adjacent trains, a collision avoidance penalty function needs to be set , the specific description is as follows:

[0122] (19)

[0123] in, is the optimal spacing adjustment coefficient; is the weight coefficient of the collision avoidance penalty function.

[0124] Therefore, the final reward function of the virtual train formation can be designed as follows:

[0125] (20)

[0126] By combining the learned model, that is, formula (11) and the final reward function, that is, formula (20), a collaborative tracking controller based on a data-driven model is proposed to achieve the collaborative control task of virtual marshaling trains. Specifically, it solves the finite prediction time domain according to formula (11) and various operating constraints. Control sequences within , and then use the learned neural network dynamics model to predict the future state. The problem is specifically described as follows:

[0127] (twenty one)

[0128] (21a)

[0129] (21b)

[0130] (21c)

[0131] (21d)

[0132] (21e)

[0133] (21f)

[0134] (21g)

[0135] For parameters and The terminal constraint set.

[0136] Due to the nonlinear characteristics of train dynamics and reward function, it is difficult to calculate the above equations accurately. Other optimization algorithms can be used to obtain approximate solutions. This application will use the Cross Entropy Method (CEM) to solve this nonlinear problem. This type of optimization algorithm can randomly generate prediction time domain The algorithm uses the learned neural network model to predict the corresponding state sequence based on the optimal predictive control sequence. The reward values ​​for all state sequences are calculated, and the action sequence corresponding to the highest expected cumulative reward is selected. This operation implements the basic principle of model predictive control, namely that the algorithm only executes the first action at each time step and receives updated state information in a timely manner. If the next time step is reached, the algorithm recalculates the optimal action sequence and repeats the above process.

[0137] CEM is an evolutionary strategy method that randomly samples from a parameterized probability distribution and updates the parameters in the distribution based on the results of each sampling, so that the probability of an action sequence with a higher cumulative reward in the distribution is higher.

[0138] This method aims to find the optimal (or suboptimal) solution for the cumulative reward function, as described below:

[0139] (twenty two)

[0140] in, is the selected optimal (suboptimal) solution vector; The fitness function of this application selects the cumulative reward function shown in formula (21), and the best (second) best solution vector corresponds to the control sequence . For any solution set.

[0141] Assumptions From the parameterized distribution The samples collected in ,and Independent of each other; is the total number of samples collected. , so that the sampled data has a set of high-value samples , is any real number. The specific description is as follows:

[0142] (twenty three)

[0143] in, For the Collect samples; for The corresponding fitness function.

[0144] Based on this set of high-value samples, a new probability distribution function can be learned But if The value of is too large. will contain only a few examples, or perhaps even none, making learning impossible. If it is set too small, the optimization process will slowly converge to the optimal (or suboptimal) solution. Therefore, in order to ensure the efficiency of the algorithm, a ratio is selected. , and Adjust to the best This corresponds to setting , provided that the samples are sorted in descending order of their values. The best samples are called elite samples. In the actual solution process, Usually a value in the range of [0.01, 0.1] is selected. Therefore, the probability distribution parameter of the elite sample set is and The calculation formula can be described as follows:

[0145] (twenty four)

[0146] are all probability distribution parameter sets of elite sample sets; 、 、 are all subsets thereof; For the i The first sample collected j A solution. i For the serial number.

[0147] (25)

[0148] In order to ensure the accuracy of the learned probability distribution function, a learning step parameter is introduced , the probability distribution parameters to be learned and Expressed as:

[0149] (26)

[0150] (27)

[0151] are all sets of probability distribution parameters for learning; 、 、 are all subsets thereof; is the corresponding probability distribution parameter set j subsets.

[0152] For clarity, the specific algorithm steps are summarized as follows:

[0153] parameter: , , iteration time ;

[0154] Initialize probability distribution parameters and : , ;

[0155] step:

[0156] For do;

[0157] For do;

[0158] From the parameterized distribution Mid-term collection samples ;

[0159] Evaluate the fitness function of each sample ;

[0160] The fitness function value of each sample Sorting;

[0161] Setting the elite threshold ; is the value corresponding to the elite threshold.

[0162] Get the elite sample set formula (23);

[0163] Obtain elite probability distribution parameters according to formula (24) and formula (25);

[0164] Update parameters according to formula (26) and formula (27) and parameters ;

[0165] End the loop.

[0166] as well as are all probability distribution parameters A subset of are all probability distribution parameters A subset of .

[0167] Finally, a simulation experiment was carried out using the laboratory's high-speed EMU tracking operation semi-physical simulation platform. The experimental results show that the learned data-driven DNN has high matching accuracy. At the same time, the reinforcement learning predictive control method can realize the coordinated tracking operation of virtual trains.

[0168] To verify the effectiveness of the proposed deep reinforcement learning data-driven model prediction algorithm for the coordinated operation control of virtual train formations, this application conducted simulation experiments using a laboratory-equipped hardware-in-the-loop simulation platform for high-speed EMU tracking operations. The experiments consisted of two parts: the first set of experiments learned a DNN model of the train based on historical operating data of sampled trains and verified the effectiveness of the data-driven model; the second set of experiments performed flexible interval formation control on two unit trains under different operating scenarios to verify the effectiveness of the control algorithm for virtual train formation control. The specific parameters of the control system are shown in Table 1.

[0169] Table 1 Control system parameters

[0170]

[0171] Experiment 1: DNN training.

[0172] The experiment uses a DNN model with two fully connected layers, and each layer contains 500 neurons. During the DNN model training process, the rectified linear units (RELU) function is selected as the activation function. , which is expressed as follows:

[0173] (28)

[0174] in, It is the function variable and has no specific meaning.

[0175] Based on the historical running data of trains running from one place to another, such as Figure 9 As shown in Figure 2, we learn the DNN model “offline” and verify the effectiveness of the trained data-driven model.

[0176] Figure 10 、 Figure 11 They are the speed prediction results and the predicted speed error, respectively. It can be seen that the error between the predicted speed output by the DNN model designed in this application and the actual speed of the train can be guaranteed to be within the range of [-0.0706, 0.0749]. Figure 12 、 Figure 13 They are the displacement prediction results and the predicted displacement error respectively. It can be seen that the error between the model output predicted displacement and the actual displacement of the train can be guaranteed to be within the range of [-14.1046, 9.6591], which verifies the effectiveness of the model.

[0177] In order to analyze the performance of DNN more intuitively, the velocity mean absolute error (MAEv), displacement mean absolute error (MAEp), velocity root mean square error (RMSEv) and displacement root mean square error (RMSEp) of the training model are calculated according to formula (29)-formula (32) to verify the effectiveness of the model. ; ; ; The results show that the proposed DNN model can estimate the high-speed train model well, verifying the effectiveness of the model.

[0178] (29)

[0179] (30)

[0180] (31)

[0181] (32)

[0182] is the train speed output by DNN; is the train displacement output by DNN; is the actual running speed of the train; is the actual running displacement of the train.

[0183] This set of experiments learned the train's DNN model "offline". In subsequent experimental simulations, the DNN model will be further learned "online" in combination with the train's online operation data. Since the learning method is the same, it will not be described further.

[0184] Experiment 2: Flexible formation control.

[0185] After verifying the prediction performance of the DNN, the train controller performance will be simulated and analyzed, assuming that the convoy consists of two trains and ignoring the impact of communication delay. This section sets up two train operation scenarios: (1) Train 1 runs at a constant speed of 250 km / h on a certain section of the road, and Train 2 cooperates with Train 1 at an initial speed of 260 km / h to achieve a formation operation state. The initial distance between the two trains is 1.2 km; (2) Train 1 runs at a variable speed along a set reference speed curve on a certain section of the road, and Train 2 cooperates with Train 1 at an initial speed of 310 km / h and performs dynamic interval formation operation control. The initial distance between the two trains is 1.6 km. The specific description of the train initial state parameters is shown in Table 2.

[0186] Table 2 Train initial conditions

[0187]

[0188] Figures 14 to 21 These are the simulation results for scenario 1 and scenario 2 respectively. Figure 14-15 、 Figure 18-19 The figures show the running status tracking of the two trains in scenario 1 and scenario 2 respectively. It can be seen that in both scenarios, the rear train can track the front train in a relatively short time and maintain the coupled running state, which verifies the effectiveness of the algorithm mentioned in this application. Figure 16 、 Figure 20 The following table shows the tracking errors of the two trains in scenario 1 and scenario 2, respectively. DAS and DAO are the errors between the actual spacing, the safe spacing, and the optimal spacing, respectively. SD is the speed error between the two trains. It can be seen that after 600s, the tracking error between the two trains in scenario 1 is approximately 0, and coupled operation of the trains can be achieved. In scenario 2, due to the influence of frequent speed changes of the trains, the tracking error will have some small fluctuations, but coupled operation of the trains can still be maintained.

[0189] Figure 17 、 Figure 21 The acceleration changes of the two trains in scenario 1 and scenario 2 respectively. It can be seen that the variation range is guaranteed to be within the specified value to meet the requirements of passenger comfort. In the initial stage, due to the limitations of the optimization algorithm, the suboptimality of the solution will be affected, resulting in some fluctuations, but the control amount is still guaranteed to be between [-1,1].

[0190] In an exemplary embodiment, a formation cooperative tracking control device is provided, comprising:

[0191] The data acquisition module is used to obtain the operating data of the virtual marshaling train; the virtual marshaling train is a coupled train cluster composed of multiple single-particle trains.

[0192] The output module is used to input the operating data into the deep neural network dynamics model to obtain output state data; the deep neural network dynamics model is obtained by using historical operating data and corresponding output state data as offline data sets to train the deep neural network.

[0193] The objective function determination module is used to determine the elastic optimal control objective function of the virtual train formation based on the elastic interval evaluation model according to the output state data using the reinforcement learning method; the elastic interval evaluation model is a mathematical model for collaborative operation based on the calculation of the safe following distance based on the relative braking distance between two adjacent trains to set the interval following distance.

[0194] The solution module is used to solve the elastic optimal control objective function using the cross entropy algorithm to obtain a control sequence; the control sequence is used to perform formation collaborative tracking control on the virtual marshaling train.

[0195] In an exemplary embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0196] In an exemplary embodiment, a computer-readable storage medium is provided, storing a computer program. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0197] In an exemplary embodiment, a computer program product is provided, including a computer program. When the computer program is executed by a processor, the steps in the above method embodiments are implemented.

[0198] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0199] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0200] This application uses specific examples to illustrate the principles and implementation methods of this application. The description of the above examples is only intended to help understand the method and core concept of this application. At the same time, for those skilled in the art, based on the concept of this application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. A formation cooperative tracking control method, characterized in that: The formation cooperative tracking control method includes: Acquiring operation data of a virtual marshaled train; the virtual marshaled train is a coupled train cluster composed of multiple single-point trains; Inputting the operating data into a deep neural network dynamics model to obtain output state data; the deep neural network dynamics model is obtained by training the deep neural network using historical operating data and corresponding output state data as an offline data set; A reinforcement learning method is used to determine the elastic optimal control objective function of the virtual marshaling train based on the output state data and a flexible interval evaluation model; the flexible interval evaluation model is a mathematical model for cooperative operation based on calculating a safe following distance based on the relative braking distance between two adjacent trains to set a following distance; The elastic optimal control objective function is solved by using a cross entropy algorithm to obtain a control sequence; the control sequence is used to perform formation cooperative tracking control on the virtual marshaled train; Actual running distance of trains and minimum safety distance The calculation formula is as follows: ； in, is the body length of the preceding train; for Common braking distance of trains following the specified time; for Emergency braking distance of the train moving forward at any moment; for The distance travelled by the train during the instantaneous reaction time; It is the parking safety margin; For trains displacement; Design optimal tracking spacing for: ； in, is the optimal spacing adjustment coefficient; the elasticity between adjacent trains Designed to: 。 2. The formation cooperative tracking control method according to claim 1, characterized in that: The method for determining the deep neural network dynamics model specifically includes: Obtain offline dataset; Normalizing the offline data set to obtain a processed data set; Build deep neural networks; Inputting the processed data set into the deep neural network, using the ADAM algorithm to train the parameters of the deep neural network with the goal of minimizing the loss function, to obtain a trained deep neural network; the parameters include: weights; The trained deep neural network is determined as a deep neural network dynamics model.

3. The formation cooperative tracking control method according to claim 2, characterized in that: The offline data set is normalized using the minimum-maximum normalization method to obtain a processed data set; the expression of the processed data set is: ； in, is the processed dataset; It is an offline dataset; is the minimum value in the offline dataset; and These are all sample numbers in the offline dataset; is the total number of samples in the offline dataset; is the maximum value in the offline dataset.

4. The formation cooperative tracking control method according to claim 2, characterized in that: The expression of the loss function is: ； in, is the loss function; is the sampling time; is the sampling time; For trains In the The operating status at the sampling time; For trains In the The operating status at the sampling time; For trains In the Control input at the sampling time; For learning models; 、 are all weight coefficients of deep neural networks.

5. The formation cooperative tracking control method according to claim 1, characterized in that: The elastic optimal control objective function specifically includes: ； in, is the elastic optimal control objective function; For the prediction time domain; is the sampling time; To predict any moment in the time domain; for Time Train The reward function of for Time Train The operating status of for Time Train control input.

6. The formation cooperative tracking control method according to claim 1, characterized in that: The cross entropy algorithm is used to solve the elastic optimal control objective function to obtain a control sequence, which specifically includes: Determine initial parameters; the initial parameters include: setting an iteration time, a ratio, and N train samples; the train samples are collected through a parameterized distribution based on a probability distribution function; Determining a fitness function according to the elastic optimal control objective function; Determining a fitness function value based on the fitness function according to a parameter combination at a current iteration time; the parameter combination includes: a ratio and a train sample; Determine whether the iteration stop condition is met; the iteration stop condition is that the current iteration time reaches the set iteration time or the fitness function value at the current iteration time is within the set elite threshold range; If not, the elite sample set is determined based on the fitness function value, and the parameters of the probability distribution function are updated based on the set learning step parameters, and "the fitness function value is determined based on the fitness function according to the parameter combination at the current iteration time" is returned; If so, the control sequence is determined according to the parameter combination at the current iteration time.

7. A formation cooperative tracking control device, characterized in that: The formation cooperative tracking control device includes: A data acquisition module is used to acquire the operation data of a virtual marshaling train; the virtual marshaling train is a coupled train cluster composed of multiple single-point trains; An output module is used to input the operating data into a deep neural network dynamics model to obtain output state data; the deep neural network dynamics model is obtained by training the deep neural network using historical operating data and corresponding output state data as an offline data set; an objective function determination module for determining, using a reinforcement learning method, a flexible optimal control objective function for a virtual marshaled train based on the output state data and a flexible interval evaluation model; the flexible interval evaluation model is a mathematical model for cooperative operation based on a set interval and following distance and calculating a safe following distance based on a relative braking distance between two adjacent trains; A solution module, configured to solve the elastic optimal control objective function using a cross entropy algorithm to obtain a control sequence; the control sequence is used to perform formation cooperative tracking control on the virtual marshaled train; Actual running distance of trains and minimum safety distance The calculation formula is as follows: ； in, is the body length of the preceding train; for Common braking distance of trains following the specified time; for Emergency braking distance of the train moving forward at any moment; for The distance travelled by the train during the instantaneous reaction time; It is the parking safety margin; For trains displacement; Design optimal tracking spacing for: ； in, is the optimal spacing adjustment coefficient; the elasticity between adjacent trains Designed to: 。 8. A computer device comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the formation cooperative tracking control method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the formation cooperative tracking control method according to any one of claims 1 to 6 is implemented.

10. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the formation cooperative tracking control method according to any one of claims 1 to 6 is implemented.

Images