Train control method based on meta reinforcement learning in multiple scenes
By adopting meta-reinforcement learning and deep reinforcement learning technology in train autonomous driving control, combined with GAN data enhancement, the problem of slow generation of train driving speed curves in multiple scenarios is solved, and fast and efficient driving speed curve generation is achieved, which improves the Operational efficiency and passenger experience.
Patent Information
- Application Number
- CN202411268935.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-11
- Publication Date
- 2024-11-12
AI Technical Summary
In the automatic driving control of trains, when facing changing operational needs and complex operating environments, the driving speed curve needs to be regenerated slowly and time-consuming, which affects the train operation efficiency and passenger experience.
Using a control method based on meta-reinforcement learning, the old task data collection, model training and new task curve generation modules are used to quickly generate new driving speed curves, and using deep reinforcement learning and contrast learning technology to pre-train the network to improve generalization ability. And use GAN for data enhancement in temporary parking scenarios.
The training time is significantly shortened, the model's rapid generalization ability under new tasks is improved, and the driving speed curve is quickly and efficiently generated, which improves railway transportation efficiency and passenger comfort.
Smart Images

Figure BDA0005038082280000071 
Figure BDA0005038082280000081 
Figure BDA0005038082280000082
Abstract
Description
Technical Field
[0001] The present invention relates to the field of transportation technology, and in particular to a control method for a train based on meta-reinforcement learning in multiple scenarios. Background Art
[0002] The main function of the automatic driving curve is to provide the target speed command to the online speed tracking control of the train automatic driving control system during the train operation, so that the train can track the target speed through continuous adjustment based on the line data and the target command speed. With the intelligence of big data, data-driven machine learning methods are often used, such as reinforcement learning, to obtain the automatic driving speed curve of the train.
[0003] However, in actual train automatic driving control, trains will face changing operational needs and complex operating environments. For example, it is necessary to adjust the inter-station scheduling time according to the actual passenger flow, the train temporarily stops due to interference from the front during operation, and due to the influence of climatic factors such as heavy rain and fog, temporary speed limit sections and speed limit values are added. At this time, it is necessary to adjust the automatic driving speed curve of the train according to operational needs and environmental changes to quickly adapt to different operational needs and environmental changes. Temporary scheduling time between train stations, temporary parking positions due to interference from the front during train operation, and temporary speed limit values increased due to climate influences that have appeared or been used are all considered old tasks. When encountering the above scenarios and the same task parameters, the driving speed curve stored in the computer can be directly called to provide the train with target speed instructions to achieve rapid control of the train.
[0004] However, when encountering new tasks that have not yet appeared, such as new temporary dispatch time, temporary speed limit value and temporary parking position, since there is no corresponding existing driving speed curve available, it is necessary to recalculate to generate a new driving speed curve each time. The biggest problem with regenerating the driving speed curve is that the generation speed is slow and time-consuming, which not only brings efficiency and economic losses to the railway transportation department, but also brings psychological irritability to passengers, reducing the riding experience. Summary of the invention
[0005] In view of the problems of the background technology, the present invention provides a control method for a train based on meta-reinforcement learning in multiple scenarios to solve the problem in the prior art that when a train needs to re-plan a driving speed curve in multiple scenarios such as temporary scheduling time adjustment, temporary speed limit or temporary stop, the driving speed curve is generated slowly, time-consuming and inefficient, which affects the normal operation of the train and causes poor passenger experience.
[0006] To achieve the purpose of the present invention, the present invention provides a control method of a train based on meta-reinforcement learning in multiple scenarios, the innovation of which is that it includes an old task data collection module, an old task model training module, a new task curve generation module and a train control module;
[0007] The control method comprises:
[0008] Assume that a train runs on a section between two adjacent stations of a certain line, and record the section as a target sub-line;
[0009] Assume that the three scenarios involved when the train needs to regenerate the driving speed curve when running on the target sub-line include: scheduling time adjustment, temporary speed limit and temporary stop;
[0010] 1) The old task data collection module collects three driving speed curve data sets generated by the train under the old task according to method 1, and then the old task data collection module transmits the three driving speed curve data sets to the old task model training module; the three driving speed curve data sets correspond to the three scene types respectively;
[0011] 2) The old task model training module is trained according to method 2 to obtain three usable models; then the old task model training module transmits the three usable models to the new task curve generation module;
[0012] 3) The new task curve generation module loads the three available models;
[0013] When the train is in one of the scenarios described in 3 and needs to re-acquire the driving speed curve:
[0014] 1) The train control module obtains the current scene type and new task parameters of the train, and then transmits the current scene type and new task parameters of the train to the new task curve generation module;
[0015] 2) Then the new task curve generation module generates a new driving speed curve by controlling the available model corresponding to the current scene type according to the new task parameters in method three;
[0016] Then the new mission curve generation module transmits the new driving speed curve to the train control module, and the train control module controls the train operation according to the new driving speed curve;
[0017] The old task refers to generating a driving speed curve according to the old task parameters in the scenario of scheduling time adjustment, temporary speed limit or temporary parking; the new task refers to generating a driving speed curve according to the new task parameters in the scenario of scheduling time adjustment, temporary speed limit or temporary parking; the parameters of both the new task parameters and the old task parameters include temporary scheduling time, temporary speed limit value or temporary parking position, and the temporary scheduling time, temporary speed limit value or temporary parking position in the new task parameters are different from the values of the corresponding parameters in the old task parameters;
[0018] The method 1 comprises:
[0019] A1) the old task data collection module collects driving speed curve data generated under multiple old tasks, and then performs image processing on all the collected driving speed curve data to obtain multiple driving speed curves;
[0020] A2) The old task data collection module classifies the plurality of driving speed curves according to the following methods for the three scene types, and obtains three driving speed curve data sets:
[0021] For the scenario of scheduling time adjustment: multiple driving speed curves generated due to temporary scheduling time adjustment are classified according to different temporary scheduling times; if there are multiple temporary scheduling times, and multiple driving speed curves are generated under each temporary scheduling time, then the multiple driving speed curves generated under the same temporary scheduling time are recorded as a first data subset, and multiple first data subsets constitute a first driving speed curve data set;
[0022] For the temporary speed limit scenario: multiple driving speed curves generated due to the temporary speed limit are classified according to different temporary speed limit values; if there are multiple temporary speed limit values, and multiple driving speed curves are generated under each temporary speed limit value, then the multiple driving speed curves generated under the same temporary speed limit value are recorded as a second data subset, and multiple second data subsets constitute a second driving speed curve data set;
[0023] For the temporary parking scenario: the temporary parking positions are divided into five categories from near to far according to the distance between the temporary parking point and the end point of the target sub-route: near, relatively close, middle, relatively far and far. Multiple driving speed curves are generated corresponding to each category of temporary parking positions. The multiple driving speed curves generated under the same category of temporary parking positions are recorded as a third data subset. Five third data subsets constitute the third driving speed curve data set.
[0024] The second method comprises:
[0025] The old task model training module constructs three DQN deep reinforcement learning models, each of which includes an encoder and a Q network constructed by a deep neural network, and the three DQN deep reinforcement learning models correspond one to one with the three driving speed curve data sets; the old task model training module uses the three driving speed curve data sets to train the three deep reinforcement learning models respectively to obtain three usable models, and the three usable models include a first usable model, a second usable model and a third usable model, and the first usable model, the second usable model and the third usable model correspond to the three scenario types of scheduling time adjustment, temporary speed limit and temporary parking respectively;
[0026] Among them, a single available model is obtained as follows:
[0027] Suppose the i-th driving speed curve dataset is any one of the three driving speed curve datasets, suppose the i-th driving speed curve dataset has n data subsets, and suppose there are m driving speed curves in the n data subsets; suppose the i-th driving speed curve dataset corresponds to the i-th DQN deep reinforcement learning model; i is an integer from 1 to 3;
[0028] B1) selecting multiple driving speed curves from the same data subset to form a first sample set, and selecting one driving speed curve from each of the n data subsets to form a second sample set;
[0029] B2) the encoder of the i-th DQN deep reinforcement learning model processes the first sample set and the second sample set using a contrastive learning method, and optimizes and updates the parameters of the encoder according to the contrastive loss function to obtain an optimized encoder;
[0030] B3) The i-th DQN deep reinforcement learning model uses a meta-reinforcement learning method to train and optimize the parameters θ of the i-th DQN deep reinforcement learning model according to the old task parameters corresponding to the m driving speed curves, and obtains the i-th available model; wherein the i-th DQN deep reinforcement learning model uses an optimized encoder to extract features from the m driving speed curves to obtain a feature representation of each corresponding old task, and then the i-th DQN deep reinforcement learning model uses the obtained feature representation of a single old task as the state input during meta-reinforcement learning training;
[0031] The third method comprises:
[0032] Assume that the available model corresponding to the scenario type of the new task is the i-th available model; the new task curve generation module calls the i-th available model, and then fine-tunes the parameters ω of the Q network using the model parameters θ of the i-th available model according to the new task parameters, and then the new task curve generation module uses the deep reinforcement learning method to generate a new driving speed curve.
[0033] Furthermore, the deep neural network described in method 2 is a CNN network.
[0034] Furthermore, when the number of driving speed curves in the third data subset is insufficient, the GAN algorithm is used to expand the number of driving speed curves in each third data subset.
[0035] The principle of the present invention is as follows:
[0036] In the existing technology, when the train needs to adjust the automatic driving speed curve to control the remaining section due to operational needs and environmental changes, machine learning methods such as meta-reinforcement learning are often used to regenerate the driving speed curve. Not only is the calculation speed slow and inefficient, but the existing model-independent meta-reinforcement learning methods such as MAML have high second-order gradient calculation complexity and overfitting problems, while the typical probabilistic embedded actor-critic meta-reinforcement learning PEARL method has problems such as difficult to ensure context quality, meta-overfitting, and unsuitability for memory-based tasks. What's more, trains face a variety of scenario tasks such as scheduling time adjustment, temporary speed limit and temporary parking, with complex task parameters and diverse parameter characteristics. It is difficult to efficiently solve practical problems using a single meta-reinforcement training.
[0037] The present invention solves the above-mentioned multi-scenario train fast and efficient optimization driving speed curve deployment and optimization problem from two aspects:
[0038] First, the present invention collects existing driving speed curves of old tasks and visualizes them, and then pre-trains the deep reinforcement learning convolutional network by comparing feature representations in an unsupervised manner, which is used to extract features and summarize multiple old tasks that have been visualized, so as to optimize the feature representation ability of the network, thereby realizing the commonality extraction of multiple old task driving strategies. On this basis, the old tasks are further pre-trained with meta-reinforcement learning to improve the generalization ability of the model. In fact, the pre-trained network can obtain more essential and universal knowledge logic in the driving curve data of each old task, and this knowledge logic represents the common knowledge of each data. Through meta-reinforcement learning pre-training, the machine learns to learn and summarize the old tasks, making the machine closer to human intelligence, so as to adapt to new tasks quickly and accurately. When facing new tasks, the deep reinforcement learning model after pre-training of the old tasks will be generalized to the deep reinforcement learning model of the new tasks, so that the new task reinforcement learning model has fast and accurate adaptability and generalization ability after a small amount of training and only a few iterative updates, that is, it can quickly and efficiently generate driving speed curves that meet the needs of new tasks. Thereby improving the problem of long training time for deep reinforcement learning.
[0039] Secondly, when collecting old task data, the data is classified into three scene types, and three machine learning models are built and trained using three data sets respectively. The above method is not only conducive to the algorithm to quickly extract the common features of data in the same scene when training old tasks, but also because the amount of data in each type of data set is more concentrated after the training data is classified, the pre-training efficiency of each model is also higher. On the other hand, when the model encounters a new task, due to the lightweight and more targeted feature representation of a single model, the model corresponding to the scene type is used to generate a new driving speed curve, which is not only efficient but also greatly improves the accuracy.
[0040] As an optimization, in the case where there may be fewer data samples for temporary parking scenarios, the present invention uses a GAN network to achieve data enhancement, thereby expanding sample data and improving model training accuracy.
[0041] It can be seen that the present invention has the following beneficial effects: under changing operational requirements and complex operating environments, when a train faces new tasks that have never appeared before, the learning method of the present invention can greatly shorten the training time and realize the function of quickly and accurately generalizing the model to new tasks, thereby realizing fast and efficient generation of train driving speed curves, which can not only improve the efficiency of the railway transportation department and reduce economic losses, but also improve the riding comfort of passengers (reduce the passenger line detention time). DETAILED DESCRIPTION
[0042] The present invention will be further described below in conjunction with the embodiments.
[0043] The hardware involved in the control method of a train based on meta-reinforcement learning in multiple scenarios of the present invention includes an old task data collection module, an old task model training module, a new task curve generation module and a train control module; the control method includes:
[0044] Assume that a train runs on a section between two adjacent stations of a certain line, and record the section as a target sub-line;
[0045] Assume that the three types of scenarios involved when the train needs to regenerate the driving speed curve when running on the target sub-line include: scheduling time adjustment, temporary speed limit and temporary stop; scheduling time adjustment is mainly caused by changes in actual passenger flow, and the departure density is adjusted by scheduling time to meet passenger demand and improve operational efficiency; temporary speed limit is mainly set in certain fixed sections between stations, such as downhill and curves. In bad weather, the speed limit values of these sections need to be adjusted; and temporary stop is mainly caused by ground equipment failure or abnormal operation of the train ahead.
[0046] 1) The old task data collection module collects three driving speed curve data sets generated by the train under the old task according to method 1, and then the old task data collection module transmits the three driving speed curve data sets to the old task model training module; the three driving speed curve data sets correspond to the three scene types respectively;
[0047] 2) The old task model training module is trained according to method 2 to obtain three usable models; then the old task model training module transmits the three usable models to the new task curve generation module;
[0048] 3) The new task curve generation module loads the three available models;
[0049] When the train is in one of the scenarios described in 3 and needs to re-acquire the driving speed curve:
[0050] 1) The train control module obtains the current scene type and new task parameters of the train, and then transmits the current scene type and new task parameters of the train to the new task curve generation module;
[0051] 2) Then the new task curve generation module generates a new driving speed curve by controlling the available model corresponding to the current scene type according to the new task parameters in method three;
[0052] Then the new mission curve generation module transmits the new driving speed curve to the train control module, and the train control module controls the train operation according to the new driving speed curve;
[0053] The old task refers to generating a driving speed curve according to the old task parameters in the scenario of scheduling time adjustment, temporary speed limit or temporary parking; the new task refers to generating a driving speed curve according to the new task parameters in the scenario of scheduling time adjustment, temporary speed limit or temporary parking; the parameters of both the new task parameters and the old task parameters include temporary scheduling time, temporary speed limit value or temporary parking position, and the temporary scheduling time, temporary speed limit value or temporary parking position in the new task parameters are different from the values of the corresponding parameters in the old task parameters;
[0054] The old task parameters or the new task parameters include temporary scheduling time, temporary speed limit value or temporary parking position, and also include the current position of the train, the position of the train operating condition conversion point, the historical average speed of the train before the current position, the running speed of the train at the current position, the slope value of the train at the current position, the starting position of the slope section of the train at the current position, the ending position of the slope section of the train at the current position, the remaining length of the train from the current position to the ending position of the slope section, the average speed of the train in the slope section at the current position, the maximum speed limit value of the train at the current position, the starting position of the maximum speed limit value, the ending position of the maximum speed limit value, the standard total running time of the train on the target sub-line, the train model, the train length, the train weight, etc.
[0055] The method 1 comprises:
[0056] A1) the old task data collection module collects driving speed curve data generated under multiple old tasks, and then performs image processing on all the collected driving speed curve data to obtain multiple driving speed curves;
[0057] A2) The old task data collection module classifies the plurality of driving speed curves according to the following methods for the three scene types, and obtains three driving speed curve data sets:
[0058] For the scenario of scheduling time adjustment: multiple driving speed curves generated due to temporary scheduling time adjustment are classified according to different temporary scheduling times; if there are multiple temporary scheduling times, and multiple driving speed curves are generated under each temporary scheduling time, then the multiple driving speed curves generated under the same temporary scheduling time are recorded as a first data subset, and multiple first data subsets constitute a first driving speed curve data set;
[0059] For the temporary speed limit scenario: multiple driving speed curves generated due to the temporary speed limit are classified according to different temporary speed limit values; if there are multiple temporary speed limit values, and multiple driving speed curves are generated under each temporary speed limit value, then the multiple driving speed curves generated under the same temporary speed limit value are recorded as a second data subset, and multiple second data subsets constitute a second driving speed curve data set;
[0060] For the temporary parking scenario: the temporary parking positions are divided into five categories from near to far according to the distance from the temporary parking point to the end point of the target sub-line: near, relatively close, middle, relatively far and far. Multiple driving speed curves are generated for each category of temporary parking positions. The multiple driving speed curves generated for the same category of temporary parking positions are recorded as a third data subset, and the five third data subsets constitute the third driving speed curve data set; since temporary parking is a low-probability event, in this scenario type, the number of posture speed curves during data collection may be too small. For this situation, the generative adversarial network GAN can be used to enhance the collected data to achieve the purpose of data expansion. For some sections of the road where there may be no driving speed curves generated due to temporary parking, sample data can be obtained by establishing line and train dynamics simulation in combination with GAN.
[0061] The second method comprises:
[0062] The old task model training module constructs three DQN deep reinforcement learning models, each of which includes an encoder and a Q network constructed by a CNN network, and the three DQN deep reinforcement learning models correspond one to one with the three driving speed curve data sets; the old task model training module uses the three driving speed curve data sets to train the three deep reinforcement learning models respectively to obtain three usable models, and the three usable models include a first usable model, a second usable model and a third usable model, and the first usable model, the second usable model and the third usable model correspond to the three scenario types of scheduling time adjustment, temporary speed limit and temporary parking respectively;
[0063] Among them, a single available model is obtained as follows:
[0064] Suppose the i-th driving speed curve dataset is any one of the three driving speed curve datasets, suppose the i-th driving speed curve dataset has n data subsets, and suppose there are m driving speed curves in the n data subsets; suppose the i-th driving speed curve dataset corresponds to the i-th DQN deep reinforcement learning model; i is an integer from 1 to 3;
[0065] B1) selecting multiple driving speed curves from the same data subset to form a first sample set, and selecting one driving speed curve from each of the n data subsets to form a second sample set;
[0066] B2) the encoder of the i-th DQN deep reinforcement learning model processes the first sample set and the second sample set using a contrastive learning method, and optimizes and updates the parameters of the encoder according to the contrastive loss function to obtain an optimized encoder;
[0067] In contrastive learning, the following contrastive loss function can be used:
[0068]
[0069] in denote the pth and qth samples selected from the first sample set and the second sample set, respectively, and f(·) denotes the encoder. The loss function l is defined as follows:
[0070]
[0071] Where ||·||2 represents the Euclidean distance, and s is the set interval used to distinguish the distance threshold of samples of different classes.
[0072] B3) The i-th DQN deep reinforcement learning model uses a meta-reinforcement learning method to train and optimize the parameters θ of the i-th DQN deep reinforcement learning model according to the old task parameters corresponding to the m driving speed curves, and obtains the i-th available model; wherein the i-th DQN deep reinforcement learning model uses an optimized encoder to extract features from the m driving speed curves to obtain a feature representation of each corresponding old task, and then the i-th DQN deep reinforcement learning model uses the obtained feature representation of a single old task as the state input during meta-reinforcement learning training; specifically:
[0073] The encoder f(·) is used to extract its feature representation z=f(x), and the extracted feature representation z is used as the state input. The model parameter θ is optimized through the meta-reinforcement learning method.
[0074] In the meta-reinforcement learning pre-training process, the driving speed curve data of different old tasks are used as different training tasks, denoted as Where k represents different tasks. Meta-learning is used to optimize model parameters so that it can quickly adjust and adapt when encountering new tasks. The objective function of meta-learning can be defined as follows:
[0075]
[0076] Among them, θ is the model parameter, α is the learning rate, For the task In each training iteration, a task is drawn from the task distribution Computational tasks The loss Update the model parameters via gradient descent: Calculate the updated loss Update the model parameters θ according to the meta-learning objective function.
[0077] Finally, the total loss function is defined by comprehensively comparing the learning loss and the meta-reinforcement learning loss:
[0078]
[0079] in For contrastive learning loss, is the meta-reinforcement learning loss, and β is the weight parameter used to balance the contribution of contrastive learning and reinforcement learning.
[0080] The third method comprises:
[0081] Assume that the available model corresponding to the scenario type of the new task is the i-th available model; the new task curve generation module calls the i-th available model, and then fine-tunes the parameters ω of the Q network using the model parameters θ of the i-th available model according to the new task parameters, and then the new task curve generation module uses the deep reinforcement learning method to generate a new driving speed curve. Specifically:
[0082] In the DQN algorithm, a Q network is designed, whose input is the state and output is the Q value of each possible action. The parameters of the Q network are initialized to the model parameters obtained in the old task pre-training to speed up the training process. At the same time, a target Q network is introduced, which has the same structure as the Q network but has slower parameter updates. It is used to calculate the target Q value, thereby stabilizing the training process.
[0083] During DQN training, experience replay technology is used to store the experience (state, action, reward, new state) of each step in a replay buffer. Small batches of samples are randomly extracted from the buffer for training to break data correlation and improve training results. The specific training process is as follows:
[0084] Initialize the Q network parameters θ and the target Q network parameters ω′, and initialize the experience replay buffer. For each training iteration, according to the current state s, select action a through the ∈-greedy strategy, that is, randomly select an action with probability ∈, and select the action that maximizes the Q value with probability 1-∈. After executing action a, observe the reward r and the next state s′ in the environment, and convert the experience (s, a, r, s′) is stored in the experience replay pool. A small batch of samples (s i , a i , r i , s′ i ), update the Q network through the following loss function:
[0085]
[0086] in, represents the sample distribution in the experience replay pool, and γ is the discount factor. By minimizing the above loss function, the Q network parameters ω are updated. And the Q network parameters are regularly copied to the target Q network, that is, ω→ω - , in order to maintain the stability of the target Q network.
[0087] Contrastive learning, deep reinforcement learning, meta-reinforcement learning, GAN network and algorithm, DQN network and algorithm, and CNN network and algorithm used in the present invention are all very common processing means or calculation methods in the prior art. Those skilled in the art can obtain relevant content from literature related to the prior art.
Claims
1. A control method for a train based on meta-reinforcement learning in multiple scenarios, characterized in that: It includes old mission data collection module, old mission model training module, new mission curve generation module and train control module; The control method comprises: Assume that a train runs on a section between two adjacent stations of a certain line, and record the section as a target sub-line; Assume that the three types of scenarios involved when the train needs to regenerate the driving speed curve when running on the target sub-line include: scheduling time adjustment, temporary speed limit and temporary stop; 1) The old task data collection module collects three driving speed curve data sets generated by the train under the old task according to method 1, and then the old task data collection module transmits the three driving speed curve data sets to the old task model training module; the three driving speed curve data sets correspond to the three scene types respectively; 2) The old task model training module is trained according to method 2 to obtain three usable models; then the old task model training module transmits the three usable models to the new task curve generation module; 3) The new task curve generation module loads the three available models; When the train is in one of the scenarios described in 3 and needs to re-acquire the driving speed curve: 1) The train control module obtains the current scene type and new task parameters of the train, and then transmits the current scene type and new task parameters of the train to the new task curve generation module; 2) Then the new task curve generation module generates a new driving speed curve by controlling the available model corresponding to the current scene type according to the new task parameters according to method three; Then the new mission curve generation module transmits the new driving speed curve to the train control module, and the train control module controls the train operation according to the new driving speed curve; The old task refers to generating a driving speed curve according to the old task parameters in the scenario of scheduling time adjustment, temporary speed limit or temporary parking; the new task refers to generating a driving speed curve according to the new task parameters in the scenario of scheduling time adjustment, temporary speed limit or temporary parking; the parameters of both the new task parameters and the old task parameters include temporary scheduling time, temporary speed limit value or temporary parking position, and the temporary scheduling time, temporary speed limit value or temporary parking position in the new task parameters are different from the values of the corresponding parameters in the old task parameters; The method 1 comprises: A1) the old task data collection module collects driving speed curve data generated under multiple old tasks, and then performs image processing on all the collected driving speed curve data to obtain multiple driving speed curves; A2) The old task data collection module classifies the plurality of driving speed curves according to the following methods for the three scene types, and obtains three driving speed curve data sets: For the scenario of scheduling time adjustment: multiple driving speed curves generated due to temporary scheduling time adjustment are classified according to different temporary scheduling times; if there are multiple temporary scheduling times, and multiple driving speed curves are generated under each temporary scheduling time, then the multiple driving speed curves generated under the same temporary scheduling time are recorded as a first data subset, and multiple first data subsets constitute a first driving speed curve data set; For the temporary speed limit scenario: multiple driving speed curves generated due to the temporary speed limit are classified according to different temporary speed limit values; if there are multiple temporary speed limit values, and multiple driving speed curves are generated under each temporary speed limit value, then the multiple driving speed curves generated under the same temporary speed limit value are recorded as a second data subset, and multiple second data subsets constitute a second driving speed curve data set; For the temporary parking scenario: the temporary parking positions are divided into five categories from near to far according to the distance between the temporary parking point and the end point of the target sub-route: near, relatively close, middle, relatively far and far. Multiple driving speed curves are generated corresponding to each category of temporary parking positions. The multiple driving speed curves generated under the same category of temporary parking positions are recorded as a third data subset. Five third data subsets constitute the third driving speed curve data set. The second method comprises: The old task model training module constructs three DQN deep reinforcement learning models, each of which includes an encoder and a Q network constructed by a deep neural network, and the three DQN deep reinforcement learning models correspond one to one with the three driving speed curve data sets; the old task model training module uses the three driving speed curve data sets to train the three deep reinforcement learning models respectively to obtain three usable models, and the three usable models include a first usable model, a second usable model and a third usable model, and the first usable model, the second usable model and the third usable model correspond to the three scenario types of scheduling time adjustment, temporary speed limit and temporary parking respectively; Among them, a single available model is obtained as follows: Suppose the i-th driving speed curve dataset is any one of the three driving speed curve datasets, suppose the i-th driving speed curve dataset has n data subsets, and suppose there are m driving speed curves in the n data subsets; suppose the i-th driving speed curve dataset corresponds to the i-th DQN deep reinforcement learning model; i is an integer from 1 to 3; B1) selecting multiple driving speed curves from the same data subset to form a first sample set, and selecting one driving speed curve from each of the n data subsets to form a second sample set; B2) the encoder of the i-th DQN deep reinforcement learning model processes the first sample set and the second sample set using a contrastive learning method, and optimizes and updates the parameters of the encoder according to the contrastive loss function to obtain an optimized encoder; B3) The i-th DQN deep reinforcement learning model uses a meta-reinforcement learning method to train and optimize the parameters θ of the i-th DQN deep reinforcement learning model according to the old task parameters corresponding to the m driving speed curves, and obtains the i-th available model; wherein the i-th DQN deep reinforcement learning model uses an optimized encoder to extract features from the m driving speed curves to obtain a feature representation of each corresponding old task, and then the i-th DQN deep reinforcement learning model uses the obtained feature representation of a single old task as the state input during meta-reinforcement learning training; The third method comprises: Assume that the available model corresponding to the scenario type of the new task is the i-th available model; the new task curve generation module calls the i-th available model, and then fine-tunes the parameters ω of the Q network using the model parameters θ of the i-th available model according to the new task parameters, and then the new task curve generation module uses the deep reinforcement learning method to generate a new driving speed curve.
2. The train control method based on meta-reinforcement learning in multiple scenarios as claimed in claim 1, characterized in that: The deep neural network described in method 2 is a CNN network.
3. The train control method based on meta-reinforcement learning in multiple scenarios as claimed in claim 1, characterized in that: When the number of driving speed curves in the third data subset is insufficient, the GAN algorithm is used to expand the number of driving speed curves in each third data subset.
Citation Information
Cited By
Intelligent rail transit signal system and automatic scheduling method thereof
CN119659715A
