Method and device for generating action decision model of virtual marshalling train

Abstract

Embodiments of the present disclosure provide a method and device for generating an action decision model of a virtual marshalling train, and are applied to the technical field of rail transit operation control. The method comprises: obtaining a set of historical state information of a target virtual marshalling train; training a centralization value network with the set of historical state information as input and a predicted acceleration vector as output; simulating the operation of the target virtual marshalling train based on the predicted acceleration vector and a pre-constructed operation environment to obtain a reward value of the predicted acceleration vector; and completing the training of the centralization value network and obtaining an action decision model of the virtual marshalling train when the reward value meets a preset condition. In this way, the centralization value network can be adjusted according to the reward value, the convergence can be accelerated, the training efficiency can be improved, and a scientific and accurate model can be provided for subsequent prediction of the next-time acceleration of the virtual marshalling train.

Description

Technical Field

[0001] This disclosure relates to the field of rail transit operation control technology, and in particular to a method and apparatus for generating motion decision models for virtual train formations. Background Technology

[0002] Compared to highway transportation, urban rail lines are characterized by shorter station spacing, frequent starts and stops, generally slower traction and braking response, and shorter cruising times. Therefore, the control methods for virtual train formation in urban rail cannot be directly applied to car platooning. Furthermore, urban rail train operation exhibits strong public transport characteristics, with a significant proportion of platform work and frequent station entry and exit processes during formation. Therefore, during virtual train formation operation, the train-to-train distance needs to be minimized during station entry and exit. Conversely, considering the safe braking distance at high speeds during exit, the train-to-train distance needs to be appropriately increased. Simultaneously, to maximize platform utilization, it is necessary to ensure synchronous starts of trains within the formation to quickly clear the platform track area.

[0003] Virtual train formations do not always maintain synchronization during inter-station operation. Instead, under certain tracking distance conditions, the operating strategies of individual trains can be appropriately adjusted to meet constraints such as impact rate and energy consumption. During the train's entry into the station, the following train needs to travel a longer distance than the preceding train (or adopt a more complex control strategy) to meet the requirement of small spacing when stopping at the station.

[0004] In the face of complex train control scenarios under virtual train formation, exploring different train control strategies to enable multiple trains to maximize punctuality and stopping accuracy while ensuring safe operation is a key issue that needs to be addressed. Summary of the Invention

[0005] This disclosure provides a method and apparatus for generating a motion decision model for a virtual train formation.

[0006] According to a first aspect of this disclosure, a method for generating a motion decision model for a virtual train formation is provided. The method includes:

[0007] Obtain the historical state information set of the target virtual train formation; wherein, the historical state information set includes historical state information at multiple times;

[0008] The centralized value network is trained by taking the historical state information set as input and the predicted acceleration vector as output.

[0009] Based on the output predicted acceleration vector and the pre-built operating environment, the target virtual train formation is simulated to obtain the reward value of the predicted acceleration vector.

[0010] When the reward value meets the preset conditions, the training of the centralized value network is completed;

[0011] The trained centralized value network is used as the motion decision model for virtual train formations.

[0012] Furthermore,

[0013] The historical state information includes observed state data and ideal acceleration;

[0014] The centralized value network includes a centralized state value network or a centralized action value network;

[0015] The process of training a centralized value network, using a set of historical state information as input and a predicted acceleration vector as output, includes:

[0016] The centralized state value network is trained by taking the observed state dataset as input and the predicted acceleration vector as output.

[0017] or,

[0018] The centralized action value network is trained by taking the observed state dataset and the ideal acceleration set as input and the predicted acceleration vector as output.

[0019] Furthermore, the target virtual train formation includes a lead car and at least one follower car;

[0020] The observation status data includes the observation status data of the lead vehicle and the observation status data of the following vehicle;

[0021] The observation status data of the navigator vehicle includes speed, acceleration, distance to the target station, and remaining running time;

[0022] The observed status data of the following vehicle includes speed, acceleration, speed of the vehicle in front, acceleration of the vehicle in front, and distance to the vehicle in front.

[0023] Furthermore, the runtime environment includes the following components:

[0024] Speed ​​limit, acceleration rate of change limit, bonus design, mask design;

[0025] The reward design items include safety rewards, parking rewards, time rewards, energy consumption rewards, and comfort rewards;

[0026] The mask design items include the mask items for the lead vehicle and the mask items for the follow vehicles.

[0027] Furthermore, the building block is constructed through the following steps:

[0028] Based on the train's traction braking characteristics, braking force braking characteristics, and road speed limit information, speed limit items are constructed.

[0029] Based on the preset absolute value of acceleration change, construct an acceleration change rate limit term;

[0030] Define corresponding domains and reward values ​​for emergency braking, collision with the vehicle in front, stopping distance, running time, running energy consumption, and absolute value of acceleration change, and construct reward design items;

[0031] Construct mask design items based on actions with negative speeds, emergency braking, and collisions with vehicles in front.

[0032] Further, the step of completing the training of the centralized value network when the reward value meets the preset conditions includes:

[0033] The reward value is compared with the preset conditions;

[0034] And when the reward value meets the preset conditions, the training of the centralized value network is completed;

[0035] When the reward value does not meet the preset conditions, the difference between the reward value and the preset threshold is calculated, the corresponding parameter value is determined based on the difference, and the parameters of the centralized value network are adjusted to continue training; wherein, the reward value is output by the reward design item of the operating environment.

[0036] According to a second aspect of this disclosure, a method for controlling the operation of virtual train formations is provided.

[0037] The method includes:

[0038] Obtain the current status information of the virtual train formation to be controlled;

[0039] Input the current state information into the motion decision model of the virtual train train trained according to any one of the methods in claims 1-6, and output the acceleration at the next moment;

[0040] Based on the acceleration at the next moment, the virtual train to be operated is controlled for operation.

[0041] According to a third aspect of this disclosure, an apparatus for generating a motion decision model for a virtual train formation is provided. The apparatus includes:

[0042] The information acquisition module is used to acquire the historical state information set of the target virtual train formation; wherein, the historical state information set includes historical state information at multiple times;

[0043] The model training module is used to train the centralized value network by taking the historical state information set as input and the predicted acceleration vector as output.

[0044] The simulation module is used to simulate the operation of the target virtual train based on the output predicted acceleration vector and the pre-built operating environment, and to obtain the reward value of the predicted acceleration vector.

[0045] The model generation module is used to complete the training of the centralized value network when the reward value meets the preset conditions; and to use the trained centralized value network as the action decision model of the virtual train formation.

[0046] According to a fourth aspect of this disclosure, an electronic device is provided. The electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the program to implement the method described above.

[0047] According to a fifth aspect of this disclosure, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the method described above.

[0048] This disclosure provides a method and apparatus for generating a motion decision model for virtual train formations. It utilizes a scalable centralized value network as a reinforcement learning model to adapt to the dynamically variable number of train formations. The centralized value network is trained based on a historical state information set. Then, based on the output predicted acceleration vector, the network is simulated in a constructed operating environment to obtain a reward value. The network's parameters are then tuned based on this reward value to accelerate convergence and improve training efficiency. The trained model predicts the acceleration of the virtual train formation at the next moment to generate an operational control strategy, ensuring safe train operation while improving punctuality and stopping accuracy.

[0049] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0050] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0051] Figure 1 A flowchart illustrating a method for generating a motion decision model for a virtual train formation according to an embodiment of the present disclosure is shown.

[0052] Figure 2 A schematic diagram of an attention-based scalable model architecture according to an embodiment of the present disclosure is shown;

[0053] Figure 3 A schematic diagram illustrating the construction process of a build item according to an embodiment of the present disclosure is shown;

[0054] Figure 4 A flowchart of a method for controlling the operation of a virtual train formation according to an embodiment of the present disclosure is shown;

[0055] Figure 5 A block diagram of an apparatus for generating a motion decision model for a virtual train formation according to an embodiment of the present disclosure is shown;

[0056] Figure 6 A block diagram of an exemplary electronic device capable of implementing embodiments of the present disclosure is shown. Detailed Implementation

[0057] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0058] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0059] This disclosure is based on deep reinforcement learning to train a multi-agent virtual train formation operation control model, which can adaptively learn and optimize control strategies to achieve automated control of virtual train formations.

[0060] Figure 1 A flowchart is shown of a method 100 for generating a motion decision model for a virtual train formation according to an embodiment of the present disclosure. Method 100 includes:

[0061] Step 110: Obtain the historical status information set of the target virtual train formation.

[0062] The historical state information set includes historical state information from multiple points in time.

[0063] In some embodiments, the target virtual train formation includes a lead car and at least one following car. Historical state information includes observed state data and ideal acceleration, wherein the observed state data includes the observed state data of the lead car and the following cars. The observed state data of the lead car includes speed, acceleration, distance to the target platform, and remaining travel time; the observed state data of the following cars includes speed, acceleration, the speed of the preceding car, the acceleration of the preceding car, and the distance to the preceding car. Real-time data such as train speed, acceleration, position information, and travel time can be collected by sensors, and the distance to the target platform and the remaining travel time can be calculated using the position and travel time information; the distance between the two cars can be directly obtained using lidar sensors, or the distance to the preceding car can be calculated by the cumulative difference in the position information observed by the two cars, etc., providing accurate and effective data support for model training.

[0064] Step 120: Train the centralized value network by taking the historical state information set as input and the predicted acceleration vector as output.

[0065] In some embodiments, historical state information includes observed state data and ideal acceleration; the centralized value network includes a centralized state value network or a centralized action value network. Training the centralized value network includes: training the centralized state value network with the observed state dataset as input and the predicted acceleration vector as output; or, training the centralized action value network with the observed state dataset and the ideal acceleration set as input and the predicted acceleration vector as output.

[0066] In some embodiments, any deep reinforcement learning algorithm can be extended to a multi-agent reinforcement learning algorithm, such as Proximal Policy Optimization (PPO) and Deep Deterministic Policy Gradient Algorithm (DDPG). Taking DDPG as an example, its features include: approximating the optimal solution using a progressive policy iteration method; combining the deep learning-based Q-learning algorithm DQN and the experience replay buffer for state updates; employing a random mini-batch sampling policy for gradient updates; and introducing a target network on top of the original Actor-Critic network framework, expanding the original two neural networks of DQN to four neural networks. However, multi-agent reinforcement learning algorithms generally follow a certain architecture, typically divided into three types: "centralized training + centralized decision-making," "decentralized training + decentralized decision-making," and "centralized training + decentralized decision-making." This disclosure does not require the use of a specific algorithm or algorithm architecture. However, the advantage of virtual grouping is that the grouping can be flexibly changed, meaning the number of groups is dynamically variable. Therefore, when using a "centralized" approach for training or decision-making, a specially designed model architecture is needed to efficiently adapt to this change. Figure 2The diagram shows a scalable model architecture based on an attention mechanism. Multiple inputs are processed through their respective multi-layer feedforward network layers to learn observation feature vectors through a front-end neural network. Then, the context vector is obtained through the attention layer. This vector is processed by the output layer to output the corresponding value.

[0067] In some embodiments, it is assumed that there are n agents in the target virtual train formation, including 1 lead car and n-1 follower cars. Each agent has its own observation state data: o 1 ...o n And ideal acceleration: a 1 ...a n Then, when training using a "centralized" approach, n centralized value networks are needed, i.e., n centralized state value networks: v([o 1 ,...,o n ];w 1 ...,v([o 1 ,...,o n ];w n ) or n centralized action value network: q([o 1 ,...,o n ],[a 1 ,...,a n ];w 1 ), ...,q([o 1 ,...,o n ],[a 1 ,...,a n ];w n ), where v is the centralized state value network, q is the centralized action value network, and w 1 ,...,w n correspond Figure 2 The parameters of the multi-layer feedforward neural network can be set to be the same or different depending on the actual situation. For example, n-1 following trains of the same model can share parameters, where n is a positive integer greater than or equal to 2. Users can choose between a centralized state value network or a centralized action value network as the model for training, without specific restrictions. By setting up two centralized training models and allowing the selection of one, the model can efficiently adapt to the dynamically variable number of train formations during training, improving the flexibility and adaptability of the model architecture.

[0068] Step 130: Based on the output predicted acceleration vector and the pre-built operating environment, perform an operation simulation on the target virtual train formation to obtain the reward value of the predicted acceleration vector.

[0069] In some embodiments, the operating environment includes the following components: speed limit, acceleration rate of change limit, reward design, and mask design; wherein the reward design includes safety reward, parking reward, time reward, energy consumption reward, and comfort reward; wherein the mask design includes a mask for the lead vehicle and a mask for the following vehicle.

[0070] In some embodiments, such as Figure 3 The diagram shown illustrates the construction process of a build item. The build item is constructed through the following steps:

[0071] Step 310: Based on the train's traction braking characteristics, braking force braking characteristics, and road speed limit information, construct speed limit items.

[0072] In some embodiments, speed limits mainly include factory speed limits and road speed limits. Factory speed limits are the train's attribute information at the time of manufacture, namely traction braking characteristics and braking force braking characteristics, which are the maximum traction acceleration and maximum braking acceleration of the virtual train formation. During train operation, its speed is limited by various factors, such as traction characteristic curves and braking characteristic curves, all of which restrict the selectable acceleration. When traction is applied, the acceleration cannot exceed the maximum acceleration allowed by the traction characteristic curve. When braking is applied, the acceleration cannot exceed the maximum braking acceleration allowed by the maximum braking curve. The formula is as follows:

[0073]

[0074] Where a is acceleration, For maximum traction acceleration, This is the maximum braking acceleration. Firstly, during operation, the speed cannot be less than 0; that is, at the current speed, the selected acceleration must ensure that the speed at the next moment is not less than 0. Secondly, the speed at the next moment cannot exceed the maximum speed limit, that is... Where v t dt represents the velocity at the current moment, and dt represents the time interval. This indicates the maximum speed at the next moment, i.e., the speed limit information of the road, the maximum speed limit of the road on which the train travels.

[0075] Step 320: Construct an acceleration change rate constraint term based on the preset absolute value of acceleration change.

[0076] In some embodiments, the rate of change of acceleration for all trains needs to be within a suitable range, and the absolute value of the rate of change of acceleration should not be too large to prevent affecting passenger comfort. Therefore, it is necessary to pre-set a threshold for the absolute value of acceleration change, that is, the absolute value of the difference between the acceleration at the current moment and the acceleration at the next moment, to limit the feasible domain of acceleration change that the train needs to follow during the simulation.

[0077] Step 330: Define the corresponding domains and reward values ​​for the emergency braking action, the collision with the vehicle in front action, the stopping distance, the running time, the running energy consumption, and the absolute value of the acceleration change, and construct the reward design items.

[0078] In some embodiments, reward values ​​are crucial for the training efficiency of reinforcement learning, i.e., centralized value networks. By setting safety, stopping, time, energy consumption, and comfort rewards for each train in the train formation and assigning corresponding reward values, the parameters of the centralized value network are adjusted using these reward values ​​to accelerate convergence and improve training efficiency while ensuring train operation safety and comfort. Specifically, this includes the following aspects:

[0079] (1) Safety bonus: For the lead car, no emergency braking is triggered throughout the journey; for the following car, in addition to not triggering emergency braking throughout the journey, a safe operating distance must be maintained relative to the car in front. For the train that triggers the corresponding action, a negative bonus can be given, and the corresponding bonus value is also relatively large.

[0080] (2) Parking Rewards: For the lead vehicle, a positive reward is given if it stops within a parking zone of ±30cm from the target position when the control command ends, and a negative reward is given otherwise; for the follow vehicle, a positive reward is given if it stops within a parking zone of -30cm from the lead vehicle when the control command ends, and a negative reward is given otherwise; correspondingly, the further away from the parking zone, the greater the reward value.

[0081] (3) Time bonus: For the lead vehicle and the follower vehicle, a positive bonus is given for completing the precise stop within the given control time, and a negative bonus is given for not stopping. Correspondingly, the further away from the given control time, the greater the bonus value.

[0082] (4) Energy consumption reward: For the lead vehicle and the follower vehicle, the energy consumption is calculated based on speed, acceleration and time interval. The lower the energy consumption, the higher the reward, and correspondingly, the smaller the reward value.

[0083] (5) Comfort bonus: For the lead vehicle and the follower vehicle, the acceleration change rate is calculated based on the acceleration and time interval. The smaller the change rate, the higher the bonus, and correspondingly, the smaller the bonus value.

[0084] Step 340: Construct mask design items based on actions with negative speed, emergency braking, and collisions with vehicles in front.

[0085] In some embodiments, for the safety of train operation, the mask design item is an action that the train absolutely prohibits. For the lead car, it is restricted to actions that cause the lead car speed to be negative and actions that cause the lead car to trigger emergency braking. For the following car, it is restricted to actions that cause the following car speed to be negative, actions that cause the following car to trigger emergency braking, and actions that cause the following car to collide with the car in front.

[0086] In some embodiments, the predicted acceleration vector obtained in step 120 is used in conjunction with the constructed operating environment to simulate the operation of the target virtual train formation and obtain the corresponding reward value.

[0087] Step 140: When the reward value meets the preset conditions, the training of the centralized value network is completed; the trained centralized value network is used as the action decision model of the virtual train formation.

[0088] In some embodiments, the reward value obtained in step 130, i.e., the reward value corresponding to each action in the train operation simulation, is compared with preset conditions. If the reward value meets the preset conditions, the training of the centralized value network is completed, and it serves as the action decision model for the virtual train formation. On the other hand, if the reward value does not meet the preset conditions, the parameters of the centralized value network need to be adjusted according to the reward value, and then training continues. For determining the parameter values, different parameter values ​​can be preset, and the corresponding parameter values ​​are determined by calculating the difference between the reward value and a preset threshold, based on the correspondence between the difference and the preset parameter values. This adjusts the parameters of the centralized value network to accelerate convergence. Simultaneously, when the trained centralized value network is used to predict the train's acceleration at the next moment and perform operation control, it achieves optimal operation control performance while ensuring safety and comfort. The preset parameter values ​​can be selected and set by the user based on experience, or they can be preset by the system and then flexibly adjusted by the user according to actual usage needs, etc.

[0089] Figure 4 A flowchart of a virtual train formation operation control method 400 according to an embodiment of the present disclosure is shown. Method 400 includes:

[0090] Step 410: Obtain the current status information of the virtual train formation to be controlled.

[0091] Step 420: Input the current state information into the motion decision model of the virtual train train trained according to any one of the methods in claims 1-6, and output the acceleration at the next moment.

[0092] Step 430: Based on the acceleration at the next moment, perform operation control on the virtual train to be operated.

[0093] In some embodiments, the motion decision model of the virtual train obtained in step 140 is used as a decision model for real-time prediction of the acceleration of the virtual train at the next moment, so as to facilitate the operation control of the virtual train based on the acceleration at the next moment.

[0094] In some embodiments, considering the dynamically variable nature of the number of train formations in virtual train sets, a centralized decision-making architecture is designed for generating operational decisions and performing operational control. This architecture is based on a centralized action network, which predicts the acceleration at the next moment. The centralized action network includes discrete and continuous types. If the current state information is discrete, the centralized action network is represented as: π([o 1 ,...,o n ];θ 1 ), ...,π([o 1 ,...,o n ];θ n If the current state information input is continuous, then the centralized action network is represented as: μ([o 1 ,...,o n ];θ 1 ), ...,μ([o 1 ,...,o n ];θ n ), where π represents: discrete centralized action network, μ represents: continuous centralized action network, [o 1 ,...,o n ] represents: the current state information input at the moment, θ 1 ,...,θ n The parameters represent the trained centralized value network. This involves selecting the parameters of the trained centralized value network to configure the parameters of the centralized action network, and then using the configured centralized action network as a decision model to predict the acceleration at the next moment. Discrete and continuous centralized action networks were designed as decision models according to practical needs, satisfying both discrete and continuous state information prediction requirements to adapt to different scenarios.

[0095] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0096] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0097] Figure 5 A block diagram of an apparatus 500 for generating a motion decision model of a virtual train formation according to an embodiment of the present disclosure is shown. Figure 5 As shown, the device 500 includes:

[0098] The information acquisition module 510 is used to acquire the historical state information set of the target virtual train formation; wherein, the historical state information set includes historical state information at multiple times;

[0099] The model training module 520 is used to train the centralized value network with the historical state information set as input and the predicted acceleration vector as output.

[0100] The simulation module 530 is used to simulate the operation of the target virtual train based on the output predicted acceleration vector and the pre-built operating environment, and to obtain the reward value of the predicted acceleration vector.

[0101] The model generation module 540 is used to complete the training of the centralized value network when the reward value meets the preset conditions; and to use the trained centralized value network as the action decision model of the virtual train formation.

[0102] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the described module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0103] According to embodiments of this disclosure, this disclosure also provides an electronic device and a readable storage medium.

[0104] Figure 6A schematic block diagram of an electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0105] Electronic device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in ROM 602 or a computer program loaded into RAM 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of electronic device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. I / O interface 605 is also connected to bus 604.

[0106] Multiple components in electronic device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of displays, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows electronic device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0107] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as method 100 and / or method 400. For example, in some embodiments, method 100 and / or method 400 may be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of method 100 and / or method 400 described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform method 100 and / or method 400 by any other suitable means (e.g., by means of firmware).

[0108] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0109] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0110] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0111] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0112] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0113] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0114] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0115] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating a motion decision model for a virtual train formation, characterized in that, include: Obtain the historical state information set of the target virtual train formation; wherein, the historical state information set includes historical state information at multiple times; A centralized value network is trained using a set of historical state information as input and a predicted acceleration vector as output; wherein, the historical state information includes observed state data and ideal acceleration; and the centralized value network includes a centralized state value network or a centralized action value network. The process of training a centralized value network, using a set of historical state information as input and a predicted acceleration vector as output, includes: The centralized state value network is trained by taking the observed state dataset as input and the predicted acceleration vector as output. or, The centralized action value network is trained by taking the observed state dataset and the ideal acceleration set as input and the predicted acceleration vector as output. Based on the output predicted acceleration vector and the pre-built operating environment, the target virtual train formation is simulated to obtain a reward value for the predicted acceleration vector; wherein, The runtime environment includes the following components: Speed ​​limit, acceleration rate of change limit, bonus design, mask design; The reward design items include safety rewards, parking rewards, time rewards, energy consumption rewards, and comfort rewards; The mask design items include the mask items for the lead vehicle and the mask items for the follow vehicles; The building block is constructed through the following steps: Based on the train's traction braking characteristics, braking force braking characteristics, and road speed limit information, speed limit items are constructed. Based on the preset absolute value of acceleration change, construct an acceleration change rate limit term; Define corresponding domains and reward values ​​for emergency braking, collision with the vehicle in front, stopping distance, running time, running energy consumption, and absolute value of acceleration change, and construct reward design items; Construct mask design items based on actions with negative speed, emergency braking, and collisions with vehicles in front; When the reward value meets the preset conditions, the training of the centralized value network is completed; The trained centralized value network is used as the motion decision model for virtual train formations.

2. The method according to claim 1, characterized in that, The target virtual train formation includes one lead car and at least one follower car; The observation status data includes the observation status data of the lead vehicle and the observation status data of the following vehicle; The observation status data of the navigator vehicle includes speed, acceleration, distance to the target station, and remaining running time; The observed status data of the following vehicle includes speed, acceleration, speed of the vehicle in front, acceleration of the vehicle in front, and distance to the vehicle in front.

3. The method according to claim 1, characterized in that, The step of completing the training of the centralized value network when the reward value meets the preset conditions includes: The reward value is compared with preset conditions; And when the reward value meets the preset conditions, the training of the centralized value network is completed; When the reward value does not meet the preset conditions, the difference between the reward value and the preset threshold is calculated, the corresponding parameter value is determined based on the difference, and the parameters of the centralized value network are adjusted to continue training; wherein, the reward value is output by the reward design item of the operating environment.

4. A method for controlling the operation of a virtual train formation, characterized in that, include: Obtain the current status information of the virtual train formation to be controlled; Input the current state information The motion decision model of the virtual train train trained according to any one of the methods in claims 1-3 outputs the acceleration at the next moment; Based on the acceleration at the next moment, the virtual train to be operated is controlled for operation.

5. A device for generating a motion decision model for a virtual train formation, characterized in that, include: The information acquisition module is used to acquire the historical state information set of the target virtual train formation; wherein, the historical state information set includes historical state information at multiple times; The model training module is used to train a centralized value network by taking a set of historical state information as input and a predicted acceleration vector as output; wherein, the historical state information includes observed state data and ideal acceleration; and the centralized value network includes a centralized state value network or a centralized action value network. The process of training a centralized value network, using a set of historical state information as input and a predicted acceleration vector as output, includes: The centralized state value network is trained by taking the observed state dataset as input and the predicted acceleration vector as output. or, The centralized action value network is trained by taking the observed state dataset and the ideal acceleration set as input and the predicted acceleration vector as output. The simulation module is used to simulate the operation of the target virtual train formation based on the output predicted acceleration vector and a pre-built operating environment, to obtain a reward value for the predicted acceleration vector; wherein, The runtime environment includes the following components: Speed ​​limit, acceleration rate of change limit, bonus design, mask design; The reward design items include safety rewards, parking rewards, time rewards, energy consumption rewards, and comfort rewards; The mask design items include the mask items for the lead vehicle and the mask items for the follow vehicles; The building block is constructed through the following steps: Based on the train's traction braking characteristics, braking force braking characteristics, and road speed limit information, speed limit items are constructed. Based on the preset absolute value of acceleration change, construct an acceleration change rate limit term; Define corresponding domains and reward values ​​for emergency braking, collision with the vehicle in front, stopping distance, running time, running energy consumption, and absolute value of acceleration change, and construct reward design items; Construct mask design items based on actions with negative speed, emergency braking, and collisions with vehicles in front; The model generation module is used to complete the training of the centralized value network when the reward value meets the preset conditions; and to use the trained centralized value network as the action decision model of the virtual train formation.

6. An electronic device, characterized in that, include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method described in any one of claims 1-4.

7. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-4.

Images