Reinforcement-based generation method for hard-to-reach scenarios of TACN RBC, generation system therefor, and RBC test system

Abstract

Disclosed in the present invention are a reinforcement-based generation method for hard-to-reach scenarios of a TACN RBC, a generation system therefor, and an RBC test system. The method comprises modeling a function of train operation dynamic system equations as a discrete-event dynamic system to describe state changes of a test environment system; in addition, the method comprises combining a Q neural network and a target neural network to implement a double-depth Q network learning algorithm, so as to construct a hard-to-reach scenario generator. The advantages are that the method obtains the hard-to-reach scenario generator on the basis of the double-depth Q network learning algorithm, effectively solving the problem that existing RBC test tools cannot find hard-to-reach scenarios, and helping to improve the quality and efficiency of RBC test tools in testing the ability of RBC systems to cope with hard-to-reach scenarios.

Description

Reinforcement generation method for difficult-to-reach scenarios of TACN RBC, and generation system and RBC test system thereof TECHNICAL FIELD

[0001] The present application relates to the field of train operation control, in particular to a reinforcement generation method for difficult-to-reach scenarios of TACN RBC, and a generation system and RBC test system thereof. BACKGROUND

[0002] In recent years, with the research and development of a new type of train operation control system (TACN system for short) based on satellite positioning, the train autonomous positioning mode using multi-source fusion and the safer train integrity checking means have broken through the limitations of traditional track circuits. The previous train tail as EOA (authorized end) adopts a speed-distance control mode, greatly shortens the train running interval, and improves the train operation efficiency.

[0003] The radio block center system (RBC for short) is a typical safety demanding system and is the core ground equipment of the TACN system. It generates and sends control commands to trains according to the messages (track section occupation, route state, temporary speed limit, etc.) received from external ground systems (interlocking equipment, temporary speed limit server, etc.) and the information (train parameters, position report, etc.) obtained from on-board equipment, and provides train authorization (MA for short) to enable trains to run safely on the lines within the jurisdiction of the RBC. The functional integrity of the radio block center equipment is the key to ensuring the safe and efficient operation of the entire system.

[0004] The radio block center system contains more than a dozen operation scenarios from registration start to parking logout. Different scenarios have different participants, interactive behaviors and control modes. There are complex dynamic switching and superposition relationships between different scenarios, which leads to the characteristics of strong concealment, diversity and complexity of potential dangerous behaviors and their causes in the radio block center system. The quality of automatic testing is based on the preparation of test scripts, but real-world scenarios have the characteristics of infinite richness, unpredictability and strong uncertainty. The test cases prepared by manual programming are constantly showing vulnerability in actual scenario applications. At the same time, in the TACN system, the train position comes from the position report of the on-board equipment, but due to the possible delay of wireless communication (maximum delay time 60s), there is a time-space lag between the train position calculated by the RBC and the actual position of the train.

[0005] In the actual operation process, the high-risk scenarios with low occurrence probability and difficult to discover in the test process are called difficult-to-reach scenarios. However, it is difficult to fully discover / generate the above-mentioned various difficult-to-reach scenarios in the existing RBC test tools to fully test the response ability of the RBC, which brings great challenges to the scheme design and verification of the RBC. Therefore, solving the problem of difficult-to-reach scenarios with small probability and high risk has become a key technical challenge for the test and verification of the radio block center system.

[0006] It can be understood that the above statements only provide background related to the present application and do not necessarily constitute prior art. SUMMARY

[0007] Based on the foregoing technical problems, the purpose of the present application is to provide a TACN RBC difficult scene reinforcement generation method, a generation system and an RBC test system, which can effectively solve the problem that existing RBC test tools cannot find difficult scenes, and help improve the test quality and efficiency of the ability of RBC test tools to cope with difficult scenes of RBC systems.

[0008] In order to achieve the above purpose, the present application is realized by the following technical scheme:

[0009] A TACN RBC difficult scene reinforcement generation method, comprising:

[0010] S1, constructing a discrete event dynamic system according to the environmental state variables of train operation;

[0011] S2, constructing a Q neural network and a target neural network through a reinforcement learning algorithm;

[0012] S3, randomly selecting a set of environmental state variable data as the current environmental state variable S t at time t;

[0013] S4, obtaining the action information ac corresponding to the current environmental state variable S t through the Q neural network;

[0014] S5, obtaining the new environmental state variable S t+1 corresponding to the next time point t+1 based on the action information ac and the discrete event dynamic system, and calculating the Q neural network reward value r t when the action information ac is implemented to the environment based on the reward function R; wherein the more the environmental state variable conforms to the difficult scene situation, the higher the reward value is when the reward function R is calculated;

[0015] S6, bringing the new environmental state variable S t+1 into the target neural network, calculating the maximum Q value of the target neural network, and obtaining the reward value y t of the S t+1 state target neural network based on the Q neural network reward value r t ;

[0016] S7, according to the Q neural network reward value r tThe Q value of the Q neural network and the maximum Q value of the target neural network are used to calculate a loss function, and the Q value deviation Δθ of the Q neural network and the target neural network is calculated, based on the loss function and the Q value deviation Δθ, network parameters θ of the Q neural network are updated;

[0017] S8, it is judged whether the first preset condition of reaching the termination state is met, if the first preset condition is not met, it is judged whether the second preset condition of updating the target neural network parameters θ - is met, if the second preset condition is met, the target neural network parameters θ - are updated, and then the process returns to S3; if the second preset condition is not met, the process returns to S3;

[0018] S9, if the first preset condition is met, the training is terminated, and the RBC difficult scene generator is obtained.

[0019] Optionally, the discrete event dynamic system is represented as:

[0020] (1)

[0021] (2)

[0022] Wherein, x(t) is a current environment state variable S t corresponding to the current time t, u(t) is a system input, which can be an action information ac, y(t) is a system output, f( ) is a processing process function, and g( ) is a state conversion operation process function.

[0023] The state equation corresponding to the discrete event dynamic system is shown in formula (3) and formula (4):

[0024] (3)

[0025] (4)

[0026] Wherein, x(t+1) is a new environment state variable S t+1 corresponding to the next time point t+1; A is a state matrix of the discrete event dynamic system, used to reflect the coupling relationship between the state variables in the system; B is an input matrix, used to reflect how the system input affects the state variables; C is an output matrix, indicating the conversion relationship from the state variables to the output; sec represents a section state, route represents a route state, traincom represents a train communication state, trainpos represents a train position state, and trainMA represents a train authorization state.

[0027] Optionally, the environment state variable includes:

[0028] at least one of a section status, a route status, a train communication status, a train position status, and a train authorization status.

[0029] Optionally, the section status comprises free, occupied, clear, train envelope, and hidden car failure occupied.

[0030] The route status comprises normal, leading, unactivated, in use, unavailable, and no other train.

[0031] The train communication status comprises normal communication, disconnected, registered, unregistered, and delayed.

[0032] The train position status comprises section, train before route, approaching section, train on route, end of route section, train through route, and leading car information.

[0033] The train authorization status comprises route.

[0034] Optionally, in the S4, the double-depth Q network generates a random number, when the random number is less than a preset value, a route allocation action is randomly selected, and when the random number is greater than or equal to the preset value, an optimal action information ac output by the current Q neural network is selected.

[0035] Optionally, the randomly selected route allocation action comprises:

[0036] an action information ac is randomly selected in the Q neural network with a probability ε t exploring the environment with a probability ε, and selecting a current optimal state with a (1-ε) probability, that is, wherein ε is a greedy probability; wherein a larger value of ε is used in an early training stage to efficiently explore a train running environment, and ε is gradually reduced as the training deepens and the state continuously shifts.

[0037] Optionally, an expression of the reward function R is shown in formula (5):

[0038] (5)

[0039] wherein r1, r2, and r3 are respectively a discovery reward, an exploration failure reward, and an unreasonable allocation reward, and α1, α2, and α3 are respectively weight coefficients of the rewards.

[0040] The discovery reward is used to improve an exploration ability of a difficult-to-reach scene generator on a difficult-to-reach scene, and an expression of the discovery reward is shown in formula (6):

[0041] (6)

[0042] The exploration failure reward is used to represent whether a difficult-to-reach scene occurs in a certain period, and an expression of the exploration failure reward is shown in formula (7):

[0043] (7)

[0044] The unreasonable allocation reward is used to represent whether an unreasonable allocation occurs in a certain period, and an expression of the unreasonable allocation reward is shown in formula (8):

[0045] (8).

[0046] Optionally, the loss function = S t+1 The maximum Q value of the state target neural network + S t The reward value r of the state Q neural network t -S t The Q value of the state Q neural network.

[0047] Optionally, the Q value deviation Δθ is:

[0048] ,

[0049] Wherein, α is a deep learning rate, S, AC respectively represent a set of state information of a discrete event dynamic system and a set of selected action information, y t is a reward value of S t+1 The state target neural network.

[0050] Optionally, the reward value y t+1 of S t The state target neural network is:

[0051] If the current S t State is a terminal state, y t =r t , otherwise y t =S t+1 The maximum Q value of the state target neural network + r t , that is , wherein γ is a discount factor.

[0052] Optionally, if the S t+1 State is not a terminal state, the quadruple {S t , ac, r t , S t+1} is put into an experience replay pool, and the Q neural network selects a plurality of quadruples from the experience replay pool for training; the target neural network selects a plurality of quadruples from the experience replay pool for training.

[0053] Optionally, the target neural network parameter θ- comprising:

[0054] copying network parameters θ of the Q neural network to the target neural network, so that θ - = θ.

[0055] Optionally, the Q neural network is a fully connected network, and a ReLU activation function is used between the fully connected layers of the Q neural network; the Q neural network provides a bounded output through a tanh activation function.

[0056] Optionally, the structure of the target neural network is the same as that of the Q neural network.

[0057] Optionally, the first preset condition comprises:

[0058] the number of training times reaches a first preset number;

[0059] and / or, the training period reaches a first preset length of time;

[0060] and / or, the train enters an incorrect route.

[0061] Optionally, the second preset condition comprises:

[0062] the number of training times reaches a second preset number.

[0063] Optionally, a TACN RBC inaccessible test scene generation system comprises:

[0064] an inaccessible scene generator generated by the reinforcement generation method for inaccessible scenes of a TACN RBC;

[0065] a discrete event dynamic system, which is communicatively connected to the inaccessible scene generator.

[0066] Optionally, an RBC test system comprises:

[0067] an inaccessible scene generator generated by the reinforcement generation method for inaccessible scenes of a TACN RBC.

[0068] Compared with the prior art, the present application has the following advantages:

[0069] The method models a function of a train operation dynamic system equation as a discrete event dynamic system to describe state changes of a test environment system; meanwhile, the method combines a Q neural network and a target neural network to realize a double deep Q network learning algorithm, and then constructs a hard-to-reach scene generator. The method can effectively search / generate hard-to-reach scenes that are likely to occur in actual operation, and helps to improve the test quality and efficiency of the ability of the RBC test tool to cope with hard-to-reach scenes of the RBC system. BRIEF DESCRIPTION OF DRAWINGS

[0070] Fig. 1 is a schematic diagram of a double deep Q network learning algorithm according to the present application;

[0071] Fig. 2 is a schematic diagram of a Q neural network and a target neural network according to the present application;

[0072] Fig. 3 is a schematic diagram of an RBC route allocation scene according to the present application;

[0073] Figs. 4 to 6 are schematic diagrams of three route allocation scenes represented by discrete event dynamic systems according to the present application;

[0074] Fig. 7 is a schematic diagram of a hidden train route allocation scene of a communication train error searched by a hard-to-reach scene generator according to the present application;

[0075] Fig. 8 is a schematic diagram of a front train route allocation scene of a rear train error searched by a hard-to-reach scene generator according to the present application;

[0076] Fig. 9 is a schematic diagram of a train operation environment architecture according to the present application. DETAILED DESCRIPTION

[0077] To make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be described below in connection with the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present application.

[0078] It should be noted that, in this document, the terms "comprising," "including," "having," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Unless otherwise specified, an element defined by the phrase "comprising..." or "including..." does not exclude the presence of additional elements in the process, method, article, or terminal device that includes said element.

[0079] It should be noted that the accompanying drawings are all in a very simplified form and use non-precise ratios, and are only used to facilitate and clearly illustrate the purpose of the embodiments of the present invention.

[0080] To address the issue of low probability and high risk of unreachable scenarios in TACN RBC system testing, this invention proposes an enhanced generation method for unreachable scenarios in TACN RBC. This method is based on Discrete Event Dynamics (DEDS) and can generate an unreachable scenario generator to produce various unreachable scenarios to test the RBC system's ability to cope with unreachable scenarios. This method helps improve the RBC testing system's ability to perform rapid RBC testing and the quality of the test.

[0081] Specifically, as shown in Figures 1 and 2, the reinforcement generation method for the difficult-to-reach scenario of the TACN RBC of the present invention includes: S1, constructing a discrete event dynamic system based on the environmental state variables of train operation; S2, constructing a Q-neural network and a target neural network through a reinforcement learning algorithm; S3, randomly selecting a set of environmental state variable data as the current environmental state variable S corresponding to the current time t. t S4. Obtain the current environmental state variable S through the Q neural network. t The corresponding action information ac; S5, based on the action information ac and the discrete event dynamic system, obtain the new environmental state variable S corresponding to the next time point t+1. t+1 The Q-neural network reward value r is calculated based on the reward function R when the action information ac is applied to the environment. t Among them, when calculating based on the reward function R, the more the environmental state variable matches the difficult-to-reach scenario, the higher the reward value; S6, change the new environmental state variable S t+1 Substitute the values ​​into the target neural network, calculate the maximum Q-value of the target neural network, and then calculate the reward value r based on the Q-value of the neural network. t Get S t+1 The reward value y of a state-target neural network t S7. Based on the Q-neural network reward value r t, the Q value of the Q neural network and the maximum Q value of the target neural network are used to calculate a loss function, and the Q value deviation Δθ of the Q neural network and the target neural network is calculated, the network parameter θ of the Q neural network is updated based on the loss function and the Q value deviation Δθ; S8, it is judged whether the first preset condition of reaching a termination state is met, if the first preset condition is not met, it is judged whether the second preset condition of updating the target neural network parameter θ - is met, if the second preset condition is met, the target neural network parameter θ - is updated, and then S3 is turned to; if the second preset condition is not met, S3 is turned to; S9, if the first preset condition is met, the training is terminated, and the RBC unreachable scene generator is obtained.

[0082] It can be seen from the above that the RBC vehicle control dynamic system model and the reinforcement learning method are combined, and the reinforcement generation method of the RBC unreachable scene based on the discrete event dynamic system is proposed. Since the scenes of the train operation system are numerous and complex, an accurate expression model cannot be established based on formal derivation, therefore, the function modeling of the train operation dynamic system equation is modeled as a discrete event dynamic system model to describe the state change of the test environment system. On the other hand, the double deep Q network learning algorithm is realized based on the Q neural network and the target neural network, and then the unreachable scene generator is constructed. In the training process, the unreachable scene generator obtains the system state from the discrete event dynamic system (environment), and calculates the action information under the current state through the double deep Q network learning algorithm. The discrete event dynamic system converts the system state under the influence of the action information (such as RBC route allocation), and outputs the action information (scene) at the next moment and the Q neural network reward value r t corresponding to the action information. The loss function and the Q value deviation Δθ of the Q neural network and the target neural network are obtained based on the double deep Q network learning algorithm, and the network parameter θ of the Q neural network is updated based on the loss function and the Q value deviation. In this method, the network parameter θ of the Q neural network is updated every cycle, and the network parameter θ - of the target neural network is updated only after the preset condition is reached, which can effectively ensure the stability of the search target within a certain time range. The unreachable scene generator generated based on the above method can effectively solve the problem that the existing RBC test tool software cannot find the unreachable scene, and is helpful to improve the test quality and efficiency of the ability of the RBC test tool software to cope with the unreachable scene of the RBC system.

[0083] From the above, in the present application, the RBC train control process is modeled and analyzed based on a discrete event dynamic system, and a discrete event dynamic system state equation is designed. In the present embodiment, the search for the difficult-to-reach scene is performed taking the RBC train operation control route allocation as an example, that is, the present embodiment focuses on the influence of the RBC processing logic on the train operation system, and the action can be set as the RBC allocating the route for the train, and therefore the RBC route allocation discrete event dynamic system state equation is designed. The discrete event dynamic system abstracts the train operation environment into system states such as trains, sections, and routes. The difficult-to-reach scene generator trained based on the discrete event dynamic system can find / generate a difficult-to-reach scene (action) that can stimulate the RBC system to produce route allocation error behaviors.

[0084] Specifically, the train operation real scene is a complex dynamic system that changes over time, and has unpredictable and inexhaustible characteristics. The line data and control rules in the scene are static structures and do not change over time, and can be generated by combination. On the contrary, the train operation process is a dynamic system that changes over time, and the state combination of the system has infinite varieties, and the discrete event combination generation method cannot exhaust all parameter combinations.

[0085] The discrete event dynamic system is a dynamic system driven by asynchronous and sudden events, and the change of the state is caused by the occurrence of various events such as the appearance or disappearance of certain environmental conditions, the start or completion of system operation, etc. The state variable of the discrete event dynamic system is x(t), and at any time t, the output y(t) of the system can be calculated from the value of the state variable at t x(t) and the input u(t). Specifically, the discrete event dynamic system can be represented as:

[0086] (1)

[0087] (2)

[0088] Wherein, x(t) is the current environmental state variable S t at the current t, u(t) is the system input, which can be the action information ac here, y(t) is the system output, f() is the processing process function, and g() is the state conversion operation process function.

[0089] Specifically, when describing the train operation process with discrete event dynamic system, the system variable x(t), i.e. the environmental state variable, is a set of variables that completely describe the number of system states. For the RBC train control system, the environmental state variable includes the section state, the route state and the train state, which contains at least one of the train communication state, the train position state (absolute position and relative section position (front end position, rear end position)), the train integrity state and the train authorization state. Further, the section state contains idle, occupied, clear, train envelope and hidden train fault occupation; the route state contains normal, leading, unactivated, in use, unavailable and other trains; the train communication state contains normal communication, disconnection, registration, deregistration and delay; the train position state contains section, train in front of route, approaching section, train on route, route end section, train passing through route and front car information; and the train authorization state contains route.

[0090] It should be noted that the system input quantity u(t) is the main driving force for driving the dynamic change of the system state, and the embodiment mainly considers the influence of RBC on train operation control, and the input is: the processing logic of RBC, mainly considering whether to allocate a route to the train. y(t) is the system output quantity, and the purpose of the embodiment is to find the difficult-to-reach scene in the RBC train control scene through the difficult-to-reach scene generator, so as to test the control logic of RBC for the difficult-to-reach scene, and therefore the system output quantity is selected as the evaluation information of the train control scene, i.e. the good or bad of the selected scene, the better the selected scene meets the situation of the difficult-to-reach scene, the higher the evaluation. If the RBC incorrectly allocates the route of other trains to the train in a certain scene, or the route remains allocated to the train when the route is degraded, it is considered that the evaluation value of the scene is high, and vice versa. f() is the processing process specified in the technical specification, including: section state conversion, route state conversion, train positioning, train permission calculation and sending, train-ground communication interface specification, etc. x(t+1) represents the system state at the next time point, and the next system state is obtained according to the system input based on the current system state.

[0091] In the TACN system, RBC needs to calculate the train permission to control the train operation. In the station, RBC calculates the train permission for the train in the unit of route. The general process is: the train runs to the front of the route, the route is opened, and RBC decides whether to authorize the train to allocate the route. After the train obtains the route authorization, it enters the route, the signal machine is closed, and the route state becomes in use. After the train leaves the route, the route is unlocked. RBC needs to allocate the route according to the position relationship between the train and the route to control the train operation. As shown in FIG. 3, the environmental state variable x(t) of the RBC route allocation scene parameterization model in an embodiment mainly includes: , wherein sec represents a section state, route represents a route state, traincom represents a train communication state, trainpos represents a train position state, and trainMA represents a train authorization state.

[0092] The section shown in FIG. 3 is composed of three sections, train 1 (tr1) is in section 2, and train 2 (tr2) is in section 1. Train 1 and train 2 run through route 1 and route 2, respectively, according to the plan, and enter the tracks 1 and 2 of station 1. In this embodiment, the environment state variable x(t) is represented as: section state [sec1[occupied], sec2[occupied], sec3[cleared]]; train communication state [traincom1[registered], traincom2[registered]]; train position state [trainpos1[section], trainpos2[section]]; train authorization state [trainMA1[M3 contains route], trainMA2[M3 does not contain route]]; route state [route1[normal], route2[normal]].

[0093] The running path and range of the train are controlled by whether the RBC allocates a route, and thus the input of the system is the decision of whether the RBC allocates a route, i.e., an action. The main hazards that can exist in the running of the train controlled by the RBC through allocation of a route are embodied in: the train is allocated an un-enterable route by mistake or there is a dangerous point (another train) within the authorized range, which can cause the train to derail or collide with another train. The un-enterable route includes two levels of meanings, one is that the interlocking judges that the route is un-enterable and sends the route opening state to the RBC through the interface information, and the other is that the RBC receives the route opening state provided by the interlocking but judges that the route is un-enterable by itself, for example, there is another train on the route. In this embodiment, the difficult-to-reach scenario generator mainly looks for scenarios in which a train is allocated an incorrect route, and thus the output of the system is whether there is an incorrect route allocated.

[0094] Based on the above, the state equation of the RBC train operation permission route allocation state, i.e., the state equation corresponding to the discrete event dynamic system, is shown in formulas (3) and (4):

[0095] (3)

[0096] (4)

[0097] wherein x(t+1) is a new environment state variable S t+1; A is the state matrix of the discrete event dynamic system, which is used to reflect the coupling relationship between the internal state variables of the system, for example, the train is approaching the section, the section state is occupied, the route state is normal, and the route signal is open; B is the input matrix, which is used to reflect how the system input affects each state variable, for example, the RBC judges that the route is route available, and allocates the route for the train; C is the output matrix, which indicates the conversion relationship of the state variable to the output, for example, the state and RBC decision are brought into the train running environment to evaluate the scene; sec represents the section state, route represents the route state, traincom represents the train communication state, trainpos represents the train position state, and trainMA represents the train authorization state.

[0098] Figs. 4 to 6 are three route allocation scenes represented by discrete event dynamic systems. As shown in Fig. 4, the train is running in the section (non-approaching section), the front station route is open, there is no communication car in front of the train section, but there is a non-communication car protection area. The state variables in this embodiment: x1=[sec1, sec2, sec3], x2=route1, x3=traincom1, x4=trainpos1, x5=trainMA1, input variable: u. As shown in Fig. 5, the train is approaching the route section, the route is open, and there is no communication car and non-communication car in front of the train. State variables: x1=[sec3], x2=route1, x3=traincom1, x4=trainpos1, x5=trainMA1, input variable: u. As shown in Fig. 6, the train is on the route, the route signal is closed, and there is no other communication car and non-communication car on the route. State variables: x2=route1, x3=traincom1, x4=trainpos1, x5=trainMA1, input variable: u.

[0099] In the selection of state variables in the train driving permission route allocation scene, in addition to considering the state of the section in front of the train and the state of the route, the position relationship between the train and the route and the train authorization state are also focused on. In addition, the allocation decision of the route is not isolated, and needs to be determined in combination with the state change of the system in continuous space and time to avoid the wrong allocation of the route for the train and thus the harm.

[0100] Because there are various combinations of sections, routes, and the positional relationship between trains and routes, it is difficult to give a clear system state matrix A. Currently, research and development and test personnel manually analyze and compile in the scene design, which is difficult to cover comprehensively. Considering the powerful black box modeling capability of neural networks, the present research constructs a RBC system hard-to-reach scene generator based on a double deep Q neural network, and uses a neural network to model the mapping relationship from the input u(t) to the output y(t) at the state x(t). The model structure of the generated hard-to-reach scene generator is a neural network, and the network weights that meet the requirements during the entire training process are saved to the hard-to-reach scene library.

[0101] The following is a specific description of constructing a hard-to-reach scene generator based on a double deep Q network algorithm to search for hard-to-reach scenes.

[0102] The dynamic interaction process of the RBC route allocation processing logic and the train operation system satisfies the Markov property, and the hard-to-reach scene generation modeling problem can be constructed as a Markov decision process (MDP). The present application combines the Q learning algorithm with the deep neural network to effectively solve the decision problem with a large solution space. The double deep Q network is an improved deep Q learning algorithm, and the present application uses two independent networks (a Q neural network and a target neural network combined to form a double deep Q network) to select and evaluate route allocation, respectively. The Q neural network is used as a function approximator for action selection, and the target neural network is used to evaluate the selected action, which can effectively solve the problem of overestimation of Q values that may occur during the learning process of the deep Q network.

[0103] In the present embodiment, the input of the Q neural network is the system state, and the output is the Q value of the route allocation / non-allocation action corresponding to the system state. Considering the input variable space scale of the system state, a simple and efficient fully connected network is selected in the research, i.e., the Q neural network is a fully connected network, which fully excavates the hidden rules behind the scene information and improves the training speed under the premise of meeting the task requirements. To improve the convergence speed of the model, a ReLU activation function is used between the fully connected layers of the Q neural network, and finally a scaling layer is used for linear scaling and biasing of the output. Here, a tanh activation function is used to provide a bounded output for route allocation decision. In the present embodiment, the structure of the target neural network is the same as that of the Q neural network.

[0104] As shown in FIGS. 1 and 2 in combination, as described in S2 to S9 in the foregoing, the present application generates a RBC hard-to-reach scene generator, i.e., an intelligent agent, using a double deep Q network method. Briefly, first, initialize the Q neural network and the target neural network, initialize the Q neural network parameter θ, and let the target neural network parameter θ - =θ. The intelligent agent obtains the current environmental state variable S tThe action information ac (whether to allocate a route) corresponding to the agent is calculated by the Q neural network, and then the action information ac is implemented on the environment to obtain a new environment state variable S t+1 , and the reward r is obtained based on the reward function t The new quadruple {S t , ac, r t , S t+1} composed of the above elements is stored in the experience replay pool. S t and ac are obtained from the experience replay pool and are brought into the Q neural network to obtain the corresponding Q value Q(S t , ac; θ), S t+1 is brought into the target neural network to obtain the maximum Q value argmax ac Q(S t , ac; θ), and is calculated, and the Q value deviation Δθ of the Q neural network and the target neural network is calculated, based on the loss function and the Q value deviation Δθ, the Q neural network parameters θ are updated by neural network back propagation. As can be seen from the foregoing, in the present application, the target neural network parameters θ are not updated at every step - , but are copied to the target neural network every several steps to ensure network training stability and avoid divergence of Q value estimation, and the closed-loop structure is used to continuously update the strategy.

[0105] In the present application, there are two ways to generate the action information ac corresponding to the current environment state variable S t . Specifically, in the S4, the double deep Q network generates a random number, when the random number is less than a predetermined value, a random route allocation action is selected, and when the random number is greater than or equal to the predetermined value, the optimal action information ac output by the current Q neural network is selected.

[0106] Specifically, the random route allocation action includes: in the Q neural network, a route allocation action ac t is randomly selected with a probability ε, that is, the environment is explored with a probability ε, and the current optimal state (the allocation action with the maximum reward value) is selected with a probability (1-ε), that is, , wherein ε is a greedy probability, . In the early stage of training, a larger value of ε is used to efficiently explore the train operation environment, and as the training deepens and the state constantly shifts, the value of ε is gradually reduced to improve the utilization efficiency of the training data.

[0107] Based on the above, according to the selected route allocation action, combined with the train position, the route state, the section state and other environment state information, the next time train position, section state and train authorization state are calculated by the discrete event dynamic system environment to obtain the next system state, i.e. the new environment state variable S t+1The reward value r corresponding to the selected action is calculated by the calculation module t , r t is determined by the reward function. After the calculation is completed, if the new environment state variable S t+1 is not a terminal state, the four-tuple {S t , ac, r t , S t+1} is put into the experience replay pool so that the Q neural network and the target neural network select a plurality of four-tuple data from the experience replay pool for training. When the Q value corresponding to the Q neural network is calculated, S t and ac can be obtained from the experience replay pool and brought into the Q neural network to obtain the corresponding Q value Q(S t , ac; theta).

[0108] The reward function is an important element of reinforcement learning, and its design directly affects the learning and evolution direction of the intelligent agent, i.e., the hard-to-reach scene generator. Therefore, the reward function is the key to the success of the reinforcement learning algorithm training. In order to make the hard-to-reach scene generator generate hard-to-reach scenes with both harm and rationality, the reward function is designed from two aspects of discovery reward and unreasonable allocation reward. The discovery reward is to improve the exploration ability of the scene generator to the hard-to-reach scene, and the unreasonable allocation reward is to regulate the compliance of the scene generator action and improve the rationality of the generated scene. In this embodiment, the expression of the reward function R is shown in formula (5):

[0109] (5)

[0110] Wherein, r1, r2, r3 are discovery reward, exploration failure reward and unreasonable allocation reward respectively, and alpha1, alpha2, alpha3 are weight coefficients of each reward, and the final value can be obtained by simulation parameter adjustment.

[0111] Specifically, the discovery reward is used to improve the exploration ability of the hard-to-reach scene generator to the hard-to-reach scene. For example, RBC allocates the route of the train error, which may cause the train error to enter the route of other trains, resulting in emergency stop, derailment or collision with other trains, causing harm, indicating that the scene meets the hard-to-reach scene condition. Therefore, the route allocation error can quantify the discovery reward of the scene generator. The expression of the discovery reward is shown in formula (6):

[0112] (6)

[0113] The exploration failure reward is used to indicate whether the hard-to-reach scene occurs within a certain period. If the hard-to-reach scene generator does not have route allocation error within a certain period, the exploration fails, and a negative reward is given, indicating that the exploration fails. The expression of the exploration failure reward is shown in formula (7):

[0114] (7)

[0115] The present application encourages the occurrence of error allocation scenarios, but is more concerned about collision accidents caused by the failure of the processing logic of the RBC, and thus reduces collision accidents caused by unreasonable processing logic of the RBC. For example, the route state sent by the interlocking system is unavailable, but the route is still allocated to the train. These unreasonable route allocation scenarios should guide the exploration direction of the difficult-to-reach scenario generator through the reward function, reducing the generation of such scenarios. In the present embodiment, the unreasonable allocation reward is used to represent whether an unreasonable allocation occurs within a certain period. The expression of the unreasonable allocation reward is shown in formula (8):

[0116] (8).

[0117] where a, p, q are constants, and specific values can be selected by simulation parameters.

[0118] On the other hand, in S6, a number of group quadruples {S t , ac, r t , S t+1} can be randomly extracted from the experience replay pool to train the target neural network, and the new environment state variable S t+1 is input into the target neural network to calculate the target allocation action and the corresponding reward y t . In the present embodiment, the reward value y t+1 of the S t state target neural network is calculated by the calculation module. If the current environment state variable S t is a terminal state, then y t =r t , otherwise y t =S t+1 state target neural network maximum Q value + r t , that is, , where γ is the discount factor.

[0119] Further, in S7, the loss function is calculated according to the Q neural network reward value r t , the Q value of the Q neural network, and the maximum Q value of the target neural network, which is the deviation of the evaluation functions of the two networks, used to stimulate the neural network to tend towards the target network. The loss function = S t+1 state target neural network maximum Q value + r t -S t state Q neural network Q value, that is, the loss function is: .

[0120] The Q value deviation Δθ of the Q neural network and the target neural network is:

[0121] ,

[0122] wherein, a is a deep learning rate, S,AC respectively represent a set of discrete event dynamic system state and a set of selected action information, y t is a reward value of the S t+1 state target neural network.

[0123] Based on the above reinforcement learning algorithm, the difficult-to-reach scene generator is trained. As described in S8 and S9, the target neural network parameter θ - is updated for a specified number of steps, and the iteration number is reached, otherwise return to S3.

[0124] In this embodiment, updating the target neural network parameter θ - includes: copying the network parameter θ of the Q neural network to the target neural network, so that θ - = θ.

[0125] Further, the first preset condition includes: the training number reaches a first preset number; and / or the training period length reaches a first preset length; and / or the train enters the wrong route.

[0126] For example, in an embodiment, the RBC standard line data is used as the train operation environment, and the uplink and downlink bidirectional train operation plan is compiled, a total of 30 pairs of trains are periodically operated, and the correspondence between the special route and the train is determined, which is used to evaluate whether the RBC route allocation meets the expectation. The train-ground communication period is 6s, the communication interruption time is 60s, and the automatic trigger route mode is used for train route handling. Introduce train-ground communication timeout, section fault occupation, route section unlocking, route turnout indication and other fault conditions. The introduction of faults is to be as close as possible to the actual process of train operation, so as to search for more edge scenes. The period of each training in the training process, i.e. the first preset length, is 1440 minutes, the sampling step is 30 minutes, and the first preset number, i.e. the maximum training number, is 5000 (in order to improve the search speed and set the time scale to improve the training efficiency). The training is realized by repeatedly setting the train operation in the train operation environment, the training period and the training number are also realized by repeatedly running the operation environment, inputting the state and index to the controller, and the controller outputs the decision of route allocation according to the state, which affects the change of the train operation environment state. For each training, when the train enters the wrong route, or the training time exceeds 1440 minutes, or the training number reaches 5000, i.e. the preset iteration number, the training is ended, and a new training is started.

[0127] Further, the second preset condition comprises: the number of training times reaches a second preset number. That is, after a number of training cycles, the network parameters are synchronized to the target neural network to ensure the stability of the search target within a certain time range. In an embodiment, the second preset number is less than the first preset number.

[0128] It should be noted that the goal of the hard-to-reach scene generator trained by the present application is not to find / generate the most dangerous hard-to-reach scene, but to find / generate as many hard-to-reach scenes as possible, which is different from the application of reinforcement learning in the optimization solution field. According to the characteristics of the optimal solution of reinforcement learning, the same scenes are obtained by continuous training, and the diversity of the scenes cannot be improved. Therefore, when the reinforcement learning training converges, the present application increases the model exploration ability to improve the diversity of the generated scenes. In practical applications, a dynamic exploration coefficient (which can be reflected in: 1) dynamically adjusting the weight of the reward function; 2) dynamically adjusting the training period, sampling time, etc.) can be introduced to improve the diversity of the hard-to-reach scene generator exploration / generation scenes. The following will illustrate the advantages of this method in dynamic interactive game through the specific test scenes searched in two training processes.

[0129] As shown in FIG. 7, in an embodiment, the train estimation front end is approaching a section, the train has no authorization, there is a protection zone in the approaching section, the front route is open, and there is a hidden train between the train front end and the route signal machine (the hidden train refers to a train that is not registered with the RBC system, and the RBC system cannot perceive the specific position thereof). The hidden train enters the first section of the route, and the route state becomes in use. If the train is allocated this route at this time, it will violate the principle of in-station route blockage driving, and the authorization of the rear train will pass through the tail of the front train, resulting in the risk of train collision.

[0130] As shown in FIG. 8, in another embodiment, there are two trains preparing to depart on different tracks. After the front train enters the route turnout section, it reports the current position to the RBC. After that, the front train communication fails to send a position report to the RBC, and has actually entered the section. If the train on another track is allocated a departure route to the same section, it will cause the authorization of the rear train (tr1) to pass through the tail of the front train and cause a rear-end collision.

[0131] When the RBC system allocates routes for trains, an aggressive strategy can ensure train operation efficiency but may cause safety hazards, and too conservative can ensure safety but has low travel efficiency. The above scenarios well test the comprehensive decision-making ability of the RBC system route allocation, and provide an effective test means for the trade-off decision between efficiency optimization and safety optimization. The route allocation scenarios searched by the method can enhance the test of route allocation safety and usability, and meet the actual test requirements. This shows that the difficult-to-reach test scenario controller constructed based on the double deep Q learning algorithm of the present application provides an effective way to solve the problems of low efficiency and incomplete scene coverage of existing RBC test software generation.

[0132] The simulation test results show that the difficult-to-reach scenario generator generated by the method can effectively improve the reliability and usability of the dynamic game simulation, scene coverage rate and repeatable test ability of the difficult-to-reach scenario.

[0133] It should be noted that the difficult-to-reach scenario generator generated by the difficult-to-reach scenario reinforcement generation method of the TACN RBC has a general scene search framework, which is not limited to the route allocation problem, but can also be extended to non-communication trains, hidden car protection and train sorting, etc. to improve the quality of RBC system testing and thus improve the design quality of the RBC system. In other embodiments, the train operation state equation in the above technical solution, i.e. the state equation corresponding to the discrete event dynamic system, can describe other necessary elements and state transition processes required for the design of the train control system, so as to provide a tool for the design of the RBC system. The double deep Q learning algorithm in the above technical solution can perform learning and search tasks uninterruptedly under unsupervised conditions, so as to save the design personnel's working time, optimize the difficult-to-reach scenario generator, and thus improve the design quality of the RBC system.

[0134] Based on the same inventive concept, the present application also provides a difficult-to-reach test scenario generation system for a TACN RBC, which comprises a difficult-to-reach scenario generator generated by the difficult-to-reach scenario reinforcement generation method for a TACN RBC, and a discrete event dynamic system, which is in communication connection with the difficult-to-reach scenario generator to realize the training of the difficult-to-reach scenario generator. Based on the difficult-to-reach scenario generator and the discrete event dynamic system, the difficult-to-reach test scenario generation system for a TACN RBC of the present application has the advantage of repeatability, and the searched difficult-to-reach scenarios can be saved in a test scenario library, i.e. a difficult-to-reach scenario library, so as to be directly applied to the system test link to verify whether the RBC system has the ability to cope with difficult-to-reach scenarios after function upgrade.

[0135] Further, the discrete event dynamic system can be regarded as a train operation environment, and the hard-to-reach scene generator uses a double deep Q network (DDQN) algorithm to make intelligent decisions according to real-time state data (such as track occupation, signal state, train position and speed, etc.) of the train operation environment, and outputs an optimal train route allocation strategy (the optimal intention is to be the most suitable for the hard-to-reach scene). The train operation environment updates the system state according to the strategy output by the controller, and feeds back the reward evaluation to realize a continuous optimization search process, aiming to improve the hard-to-reach scene search and generation efficiency of the hard-to-reach scene generator.

[0136] For example, taking the receiving process shown in FIG. 1 as an example, the operation principle of the route allocation method based on the reinforcement learning principle is explained. The current environment state variable S t That is, S t State: Section state [sec1 [normal occupation], sec2 [normal occupation], sec3 [free]]; Train communication state [train1 [normal communication], train2 [normal communication]]; Train position state [train1 [train in front of route], train2 [train in front of route]]; Train authorization state [train1 [M3 does not contain route], train2 [M3 does not contain route]]; Route state [route1 [normal, no other train], route2 [not activated, no other train]].

[0137] The hard-to-reach scene generator selects the next action according to the Q network decision rule: allocates route1 for train1. Then the next state S t+1 State: Train authorization state [train1 [M3 contains route1], train2 [M3 does not contain route]]; Section state is driven by train and track circuit acquisition information; Train communication state is determined by communication condition; Route state transition is driven according to the process of train entering the route. In order to explore more hard-to-reach scenes, random occupation and clearing can be added in the section state to simulate section fault occupation and poor shunting; random communication delay is added in the train communication state to simulate the deviation between the actual position of the train and the position report. At the same time, route segment unlocking, route turnout failure and the presence of other trains are randomly added in the route state.

[0138] As shown in FIG. 9, the train operation environment is composed of a state indicator, an RBC simulation controller, a communication server, a TSRS simulator (temporary speed restriction simulator), an OBC simulator (on-board simulator), an NRBC simulator, a CBI simulator, RBC software, and SDMS (maintenance station software). Among them, the state indicator is responsible for extracting the system state from the RBC simulation controller, transmitting the route allocation decision generated by the inaccessibility scenario generator, i.e., action information ac, to the RBC software for execution (the generated decision needs to be input to the environment), and further evaluating the decision; the RBC simulation controller is responsible for editing and managing the train operation sequence and sending it to the OBC simulator, the TSRS simulator, the CBI simulator, the CTC simulator, and the adjacent RBC software for execution. The communication server is responsible for forwarding the messages of the RBC simulation controller and receiving the messages of the peripheral simulator to the RBC simulation controller. The on-board simulator, the temporary speed restriction simulator, the TSRS simulator, and the adjacent RBC software can all communicate with the lower-level RBC (real device) to assist the simulation of the train operation control system response to actions. The simulation controller obtains the train operation results through the maintenance station software SDMS, and converts them into state information and evaluation results recognized by the inaccessibility scenario generator. The state and decision are executed in the train operation environment, the decision is evaluated according to the set evaluation index, the Q neural network parameters are updated, and the Q neural network is prompted to make decisions according to the set target. The above process is repeated until the number of iterations or the convergence requirement is met.

[0139] Based on the same inventive concept, the application also provides an RBC test system containing an inaccessibility scenario generator generated by the aforementioned TACN RBC inaccessibility scenario reinforcement generation method. In actual application, the RBC test system can generate various inaccessibility scenarios based on the inaccessibility scenario generator, and then train and test the RBC system to improve the test accuracy of the RBC system's ability to cope with inaccessibility scenarios. Therefore, the RBC test system can effectively solve the problem that the existing RBC test tool software cannot find inaccessibility scenarios, and improve the software test quality and efficiency.

[0140] In summary, in the TACN RBC inaccessibility scenario reinforcement generation method, its generation system and RBC test system, the method models the function of the train operation dynamic system equation as a discrete event dynamic system to describe the state change of the test environment system; at the same time, the method combines Q neural network and target neural network to realize double-depth Q network learning algorithm, and then constructs an inaccessibility scenario generator. The method can effectively search / generate inaccessibility scenarios that may occur in actual operation, which helps to improve the test quality and efficiency of the RBC test tool's ability to cope with inaccessibility scenarios of the RBC system.

[0141] While the application has been described in detail and with reference to specific preferred embodiments thereof, it will be apparent to one skilled in the art that various modifications and alternatives can be employed without departing from the spirit and scope of the application. Accordingly, the scope of the application should be determined by the appended claims and their equivalents.

Claims

1. A method for augmented generation of hard-to-reach scenarios of TACN RBCs, characterized by, The method comprises the steps of: S1, constructing a discrete event dynamic system according to an environment state variable of train operation; S2, constructing a Q neural network and a target neural network through a reinforcement learning algorithm; S3, randomly select a set of environmental state variable data as the current environmental state variable S corresponding to the current time t t ; S4, obtaining a current environment state variable S through the Q neural network t Corresponding action information ac; S5、based on the action information ac and the discrete event dynamic system, obtaining a new environment state variable S corresponding to a next time point t+1 t+1 , based on the reward function R, calculating a Q neural network reward value r when the action information ac is implemented to the environment t ; wherein, the more the environment state variable conforms to the difficult-to-reach scene situation when the reward function R is calculated, the higher the reward value is. S6, the new environment state variable S t+1 into the target neural network, calculate the maximum Q value of the target neural network, and based on the Q neural network reward value r t acquire S t+1 reward value y of the state target neural network t ; S7、According to the Q neural network reward value r t The Q value of the Q neural network and the maximum Q value of the target neural network calculate a loss function, and calculate the Q value deviation Δθ of the Q neural network and the target neural network. Based on the loss function and the Q value deviation Δθ, update the network parameter θ of the Q neural network. S8, determine whether a first preset condition of reaching a termination state is met, if the first preset condition is not met, determine whether a second preset condition of updating the target neural network parameter θ - is met, if the second preset condition is met, update the target neural network parameter θ - , and then go to S3; if the second preset condition is not met, go to S3; S9, if the first preset condition is met, terminating the training to obtain a hard-to-reach scenario generator of the RBC.

2. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. The discrete event dynamic system is represented as: （1） （2） Wherein, x(t) is the current environmental state variable S corresponding to the current t moment t , u(t) is the system input, which can be the action information ac, y(t) is the system output, f() is the processing function, and g() is the state conversion operation process function. The state equation corresponding to the discrete event dynamic system is shown in formulas (3) and (4): （3） （4） wherein x(t+1) is a new environment state variable S corresponding to a next time point t+1 t+1 ; A is a state matrix of the discrete event dynamic system, used to reflect the coupling relationship between the state variables inside the system; B is an input matrix, used to reflect how the system input affects the state variables; C is an output matrix, indicating the conversion relationship of the state variables to the output; sec represents a section state, route represents a route state, traincom represents a train communication state, trainpos represents a train position state, and trainMA represents a train authorization state.

3. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. The environment state variable comprises: At least one of a section state, a route state, a train communication state, a train position state and a train authorization state.

4. The method of claim 3, wherein The section state comprises idle, occupied, clear, train envelope and hidden train fault occupation. The route state comprises normal, leading, inactivated, in use, unavailable and other trains. The train communication state comprises normal communication, disconnection, registration, deregistration and delay. The train position state comprises section, train in front of route, approaching section, train on route, route end section, train passing through route and front train information. The train authorization state comprises route.

5. The method of claim 1, wherein In S4, the double-depth Q network generates a random number, when the random number is less than a preset value, a route allocation action is randomly selected, and when the random number is greater than or equal to the preset value, an optimal action information ac output by the current Q neural network is selected.

6. The enhanced generation method for difficult-to-reach scenarios in TACN RBC as described in claim 5, characterized in that, The random selection of the route allocation action comprises: In the Q neural network, action information ac is randomly selected with a probability ε t With a probability of ε to explore the environment, and with a probability of (1-ε) to select the current optimal state, that is, Wherein, ε is a greedy probability; wherein, a larger value of ε is used in the early stage of training to efficiently explore the train operation environment, and ε is gradually reduced as the training deepens and the state continuously shifts.

7. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. The expression of the reward function R is shown in formula (5): （5） Wherein, r1, r2 and r3 are discovery reward, exploration failure reward and unreasonable allocation reward respectively, and α1, α2 and α3 are weight coefficients of each reward; The discovery reward is used to improve the exploration ability of the hard-to-reach scenario generator to the hard-to-reach scenario, and the expression of the discovery reward is shown in formula (6): （6） The exploration failure reward is used to indicate whether a hard-to-reach scenario occurs within a certain period, and the expression of the exploration failure reward is shown in formula (7): （7） The unreasonable allocation reward is used to indicate whether unreasonable allocation occurs within a certain period, and the expression of the unreasonable allocation reward is shown in formula (8): （8）。 8. The method of claim 1, wherein The loss function = S t+1 State target neural network maximum Q value + S t State Q neural network reward value r t -S t Q value of the state Q neural network.

9. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. The Q value deviation Δθ is: ， Wherein, a is a deep learning rate, S, AC respectively represent a set of discrete event dynamic system states and a set of selected action information, y t is S t+1 The reward value of the state target neural network.

10. The method of claim 9, wherein the TACN RBCs are generated in a difficult-to-reach scene. S t+1 reward value y of the state-target neural network t is: If the current S t state is the terminal state, then y t = r t , otherwise y t = S t+1 target neural network maximum Q value + r t , i.e. Wherein, γ is a discount factor.

11. The method of claim 1, wherein If S t+1 State is not the terminal state, put the quadruple {S t , ac, r t , S t+1} into the experience replay pool, the Q neural network selects several quadruples from the experience replay pool for training; the target neural network selects several quadruples from the experience replay pool for training.

12. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. updating target neural network parameters θ - comprising: Copy the network parameters θ of the Q neural network to the target neural network, such that θ - = θ.

13. The method of claim 1, wherein The Q neural network is a fully connected network, and a ReLU activation function is used between the fully connected layers of the Q neural network; and the Q neural network provides a bounded output through a tanh activation function.

14. The method of claim 1, wherein the first preset condition comprises: the target neural network has the same structure as the Q neural network.

15. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. the first preset condition comprises: the number of training reaches a first preset number; and / or, the training period reaches a first preset duration; and / or, the train enters an incorrect route.

16. The method of claim 1, wherein the TACN RBCs are generated in a difficult-to-reach scene. the second preset condition comprises: the number of training reaches a second preset number.

17. A system for generating a difficult-to-access test scenario for a TACN RBC, the system comprising: comprising: a hard-to-reach scenario generator generated by the method of claim 1-16; a discrete event dynamic system in communication with the hard-to-reach scenario generator.

18. An RBC test system characterized by, comprising: a hard-to-reach scenario generator generated by the method of claim 1-16.

Images