A mine unmanned motor car speed control method based on an improved epsilon-greedy strategy

Abstract

The application discloses a mine-used unmanned motor car speed control method based on an improved epsilon-greedy strategy and reinforcement learning, which comprises the following steps: taking the mine-used unmanned motor car as an intelligent agent, setting the intelligent agent action as an integer in the best driving / braking torque range applied to the axle of the motor car; designing the intelligent agent action, state and reward functions respectively; using the Q-Learning algorithm to take the real-time state information of the motor car as input, the driving / braking torque applied to the axle and the reward value obtained by the intelligent agent as output, completing the main algorithm structure building of the mine-used unmanned motor car speed control, and ensuring that the mine-used motor car autonomously completes behaviors such as safe following and effective obstacle avoidance under complex and uncertain working conditions; through the improvement of the epsilon-greedy strategy, the learning efficiency of the reinforcement learning algorithm of the mine-used motor car is effectively improved, and better control effect is achieved under fewer iteration times.

Description

Technical Field

[0001] This invention relates to the field of unmanned driving technology for coal mine auxiliary transportation equipment, and in particular to a speed control method for unmanned electric locomotives used in mines based on an improved ε-greedy strategy and reinforcement learning. Background Technology

[0002] As the most important transportation tool in the main transport roadways, electric locomotives are responsible for transporting coal, gangue, equipment, and materials in coal mine production. Traditional auxiliary transport systems involve numerous transfer links, a large number of personnel, complex personnel behavior, and relatively outdated technology and equipment. Due to the slippery conditions, narrow and enclosed spaces, numerous bends, and insufficient lighting in the roadways, there are many potential hazards, such as locomotive rear-end collisions, pedestrian injuries, and other accidents, resulting in casualties and property damage. Therefore, researching unmanned driving systems for coal mine auxiliary transport is essential for achieving intelligent and less-manned auxiliary transport in coal mines. Considering the vehicle behavior requirements under specific operating conditions such as locomotive start-up, encountering obstacles, curves, and braking, obtaining the operating speed curve of the unmanned electric locomotive is the primary problem to be solved in the research of autonomous operation control of electric locomotives. Traditional speed curve optimization algorithms have poor robustness under extreme track conditions and are prone to getting trapped in local optima. Dynamic programming methods, because they reflect the connections and characteristics of dynamic process evolution, are therefore applied to train speed curve design. Currently, train speed curve optimization design technology has become increasingly mature, and the design process often considers the train's time-saving and energy-saving performance. However, due to the limitations of industry development and the complexity of its working environment, there is little research on the speed curve design of mining locomotives. Summary of the Invention

[0003] Purpose of the invention: To address the above problems, this invention proposes a speed control method for unmanned mining locomotives based on an improved ε-greedy strategy and reinforcement learning. This method can ensure that the mining locomotive can autonomously complete behaviors such as safe following and effective obstacle avoidance in complex and uncertain working conditions. Furthermore, by improving the ε-greedy strategy, the learning efficiency of the reinforcement learning algorithm of the mining locomotive is effectively improved, achieving better control results with fewer iterations.

[0004] Technical solution: To achieve the objectives of this invention, the technical solution adopted is as follows:

[0005] A speed control method for unmanned electric locomotives used in mining, based on an improved ε-greedy policy and reinforcement learning, includes the following steps:

[0006] (1) The mining unmanned electric locomotive is used as an intelligent agent, and the intelligent agent's actions are set to integers within the optimal driving / braking torque range applied to the axle of the locomotive.

[0007] (2) Classify the state of the intelligent agent according to the driving conditions of the unmanned locomotive in the mine;

[0008] (3) Design the agent reward function by setting different reward values ​​according to the agent's actions in different states;

[0009] (4) Using the Q-Learning algorithm, the real-time status information of the locomotive is used as input, and the driving / braking torque applied to the axle and the reward value obtained by the agent are used as output to build a speed control model for the unmanned locomotive used in mining.

[0010] (5) Use the ε-greedy policy to generate actions, and learn to update the Q function based on the agent's action at time t and the state and reward value obtained from the interaction with the environment at time t+1.

[0011] (6) The reinforcement learning process is carried out within the set number of iterations. After reaching the maximum number of iterations, the speed control of the unmanned electric locomotive is completed.

[0012] Furthermore, the driving status of the unmanned mining locomotive is divided into different states based on whether it travels on curved / straight road sections, encounters obstacles, reaches its maximum speed, or reaches its destination.

[0013] Furthermore, different reward values ​​are set for the agent's actions in different states, and the reward function is set as follows:

[0014] reward = c * (v record -v current )

[0015] Among them, v record It is the vehicle speed feedback value within the previous sampling interval, v current It is the vehicle speed feedback value at the current sampling time, and c is the reward function coefficient, the sign of which is determined by whether the vehicle is speeding in different states.

[0016] Furthermore, the Q-function is learned and updated based on the agent's action at time t and its interaction with the environment, resulting in its state at time t+1 and its reward value, as detailed below:

[0017] Define the state S of the agent at time t. t and the action taken A t Define the Q function Q(S) t A t This is used to estimate the action value function q of all executable actions a in any state s when strategy π is adopted. π (s,a);

[0018] Define the Q-value of an agent at time t+1 as Q(S) when the agent takes any action to interact with the environment at time t. t+1 ,a), using the method that makes Q(S t+1 a) Update Q(S) using the action with the largest value. t A t The updated formula is as follows:

[0019] Q(S t A t )←Q(S t A t )+β[R t+1 +γmaxQ(S t+1 ,a)-Q(S t A t )]

[0020] Where β is the learning rate, γ is the discount factor, and R0 t+1 Based on state S t And Action A t The reward value obtained.

[0021] Furthermore, the action is generated using the ε-greedy strategy, specifically as follows:

[0022] The agent's driving actions are randomly obtained according to the ε-greedy policy. The expression of the ε-greedy policy is as follows:

[0023]

[0024] Among them, a t Let ε be the action of the agent at time t, A be the set of all actions, and ε be the action of the agent at time t. t Let ε be the exploration rate of the agent at time t, i.e., the probability of randomly selecting an action, 0 < ε t <1, the probability of selecting the action amount based on the maximum Q function is 1-ε t ;ε t The method for obtaining the value is as follows:

[0025]

[0026] Where, ε initial ε is the initial exploration rate. final The exploration rate is the number of iterations at the termination time, epoch is the current iteration number, and max_epoches is the maximum number of iterations.

[0027] Determine whether the difference between the action value to be taken at time t obtained according to the ε-greedy strategy and the action value at time t-1 exceeds a set threshold;

[0028] If the threshold is exceeded, the ε-greedy strategy determines whether the vehicle should apply driving torque or braking torque at time t. Based on the action taken at time t-1, acceleration / deceleration is performed at time t. If the threshold is not exceeded, the action to be taken at time t determined by the ε-greedy strategy is executed.

[0029] Beneficial effects: Compared with the prior art, the technical solution of the present invention has the following beneficial technical effects:

[0030] This invention fully utilizes the characteristics of reinforcement learning as a method for studying how intelligent agents can maximize the rewards they obtain in complex and uncertain environments, and realizes the universality of the locomotive speed control strategy in different mine roadway environments. That is, one set of control strategies can be applied in different roadways, solving the problem that existing locomotive speed control processes require continuous collection and updating of roadway environment data due to changes in the mine environment.

[0031] This invention proposes a speed control method for unmanned mining locomotives based on reinforcement learning, ensuring that the locomotives can autonomously follow other vehicles and effectively avoid obstacles in complex and uncertain mining environments. Furthermore, an improved ε-greedy strategy is proposed based on general reinforcement learning algorithms, effectively enhancing the application efficiency of reinforcement learning algorithms in unmanned speed control of mining locomotives, enabling the agent to achieve better control results with fewer iterations. This invention shows promising application prospects in unmanned speed control of locomotives in complex mining environments. Attached Figure Description

[0032] Figure 1 This is a flowchart of the speed control method for unmanned electric locomotives used in mining according to the present invention;

[0033] Figure 2 This is a schematic diagram of the speed control system for the unmanned mining locomotive of the present invention.

[0034] Figure 3 This is a comparison chart of parameter values ​​before and after the improvement of the ε-greedy strategy of this invention;

[0035] Figure 4 This is a distance-velocity relationship graph obtained by iterating the method of this invention 500 times;

[0036] Figure 5 The method of this invention iterates the distance-speed relationship and road condition map 500 times;

[0037] Figure 6 This is a comparison chart of the reward values ​​obtained by the agent before and after the improvement of the ε-greedy strategy of this invention;

[0038] Figure 7This is a comparison chart showing the difference between the following distance before and after the improvement of the ε-greedy strategy of this invention and the set standard following distance. Detailed Implementation

[0039] The technical solution of the present invention will be further described below with reference to the accompanying drawings and embodiments.

[0040] Figure 1 The flowchart below shows the speed control method for unmanned mining locomotives based on the improved ε-greedy strategy and reinforcement learning of the present invention. The specific implementation strategy is as follows:

[0041] Step 1: Configure the agent and its specific actions based on reinforcement learning algorithms.

[0042] Taking the CTY1.5 / 6 electric locomotive model as an example for mining electric locomotive implementation, and considering it as an intelligent agent, its control system includes: an environmental perception module, an intelligent decision-making module, an action execution module, a network communication module, and a power supply module. Environmental Perception Module: Without GNSS or other auxiliary positioning technologies, the mining unmanned electric locomotive integrates sensor information from LiDAR, network cameras, and ultrasonic radar to achieve environmental perception in the complex underground space of a coal mine. Intelligent Decision-Making Module: The mining unmanned electric locomotive processes the information collected by the environmental perception module through an industrial control computer and sends real-time vehicle control commands to the vehicle controller after algorithm execution, realizing intelligent decision-making during the unmanned driving process. Action Execution Module: The driving motor controller and hydraulic electromagnetic proportional valve of the mining unmanned electric locomotive receive commands from the vehicle controller and execute corresponding actions, completing vehicle speed adjustment and emergency braking. Network Communication Module: The modules communicate with each other using TCP / IP and serial communication, and also use wireless transmission to transmit vehicle underground operation information to the ground dispatch room in real time. Power Supply Module: Provides power supply for the different power needs of different modules.

[0043] Figure 2This is a schematic diagram of the speed control system for an unmanned mining locomotive. First, the locomotive's driving environment information, collected by millimeter-wave radar and ultrasonic radar, is transmitted to a serial server via RS485 and RS232 serial ports, respectively. This same signal transmission method is used for point cloud images and visual images of the locomotive's driving environment collected by lidar and network cameras, transmitting them via TCP / IP to a WIFI6 router, and then via TCP / IP to the industrial control computer. Second, the industrial control computer performs information fusion, feature extraction, and calculation to optimize the driving speed curve and sends specific instructions to the vehicle controller via the CAN bus. Finally, the drive motor controller and hydraulic electromagnetic proportional valve receive real-time instructions from the industrial control computer and the vehicle controller via the CAN bus, controlling the permanent magnet synchronous motor and braking system respectively, thereby achieving speed regulation and emergency braking of the mining locomotive. The locomotive is equipped with an onboard base station, which can upload the vehicle environment, real-time speed, and driving path information processed by the industrial control computer in real time, and receive scheduling information from the monitoring center and ground server.

[0044] When the locomotive operates under different conditions, the agent selects an appropriate action based on the current state. The locomotive's movement is a continuous process; the process is discretized with a sampling interval of 0.01 seconds to ensure the agent obtains the latest environmental information in real time and provides the optimal strategy. A dynamic model of the mining locomotive is established using Simpack software, and the optimal driving / braking torque range applied to the axle is determined to be [-300 Nm, 300 Nm]. Since this invention primarily addresses the impact of speed on the locomotive's dynamic performance, the motor and gear chain power transmission components are ignored, and the agent's actions are set to integers within the optimal driving / braking torque range applied to the axle.

[0045] Step 2: Classify the state of the intelligent agent based on the operating conditions of the unmanned electric locomotive used in mining.

[0046] Based on the driving status of the unmanned mining locomotive on curved / straight roads, whether it encounters obstacles, whether it reaches its maximum speed, and whether it reaches its destination, the vehicle's driving status is divided into different states as shown in Table 1.

[0047] Table 1. Speed ​​Control Status Settings for Electric Locomotive Reinforcement Learning

[0048]

[0049] Step 3: Set different reward values ​​for the agent's actions in different states, and design the agent's reward function.

[0050] Taking into account both vehicle operating efficiency and whether the speed meets the standard, the reward function is set as follows:

[0051] reward = c * (v record -v current )

[0052] Among them, v record It is the vehicle speed feedback value within the previous sampling interval, v current It is the vehicle speed feedback value at the current sampling time, and c is the reward function coefficient, the sign of which is determined by whether the vehicle is speeding in different states.

[0053] Step 4: Build the main algorithm structure for speed control of unmanned mining locomotives.

[0054] The Q-Learning algorithm is used as the core algorithm for the speed control method of unmanned mining vehicles. An ε-greedy policy is used to generate actions, and another optimal Q-function is learned based on the next state obtained from the interaction between the action and the environment, as well as the reward value. This learning process takes real-time vehicle state information such as vehicle speed, wheel speed, and position as input, and the driving / braking torque applied to the axles and the reward value obtained by the agent as output, performing a reinforcement learning process within a set number of iterations.

[0055] Define the state S of the agent at time t. t and the action taken A t Define the Q function Q(S). t A t This is used to estimate the action value function q of all executable actions a in any state s when strategy π is adopted. π (s,a), defining the Q-value at time t+1 as the result of an agent taking any action and interacting with the environment at time t. t+1 ,a), and adopt the method that makes Q(S) t+1 a) Update Q(S) using the action with the largest value. t A t The updated formula is as follows:

[0056] Q(S t A t )←Q(S t A t )+β[R t+1 +γmaxQ(S t+1 ,a)-Q(S t A t )]

[0057] Where β is the learning rate, γ is the discount factor, and R0 t+1 Based on state S t And Action A t The reward value obtained.

[0058] The action is generated using an ε-greedy strategy, the specific strategy of which is as follows:

[0059] First, the agent's driving action is obtained randomly or based on empirical values ​​according to the ε-greedy policy. The expression of the ε-greedy policy is as follows:

[0060]

[0061] Among them, a t Let ε be the action of the agent at time t, A be the set of all actions, and ε be the action of the agent at time t. t (0<ε t <1) represents the agent's exploration rate at time t, where ε represents the probability of randomly selecting an action. t The probability of selecting the action amount based on the largest Q function is 1-ε. t ε t The method for obtaining the value is as follows:

[0062] ε t1 =(ε initial -ε final )·(1-epoch / max_epoches)

[0063] Where, ε initial ε is the initial exploration rate. final The exploration rate is the number of iterations at the termination time, epoch is the current iteration number, and max_epoches is the maximum number of iterations.

[0064] Secondly, in order to ensure the smoothness of the driving control speed of the unmanned locomotive in the mine and to prevent the vehicle from malfunctioning due to sudden speed being too low or too high, it is determined whether the difference between the action value to be taken at time t obtained according to the ε-greedy strategy and the action value at time t-1 exceeds the set threshold.

[0065] Finally, if the threshold is exceeded, the ε-greedy strategy determines whether the vehicle should apply driving torque or braking torque at time t. Based on the action taken at time t-1, acceleration / deceleration is performed at time t (adding / subtracting the threshold). If the threshold is not exceeded, the action to be taken at time t determined by the ε-greedy strategy is executed.

[0066] Step 5: Improve the ε-greedy strategy.

[0067] The method of determining the value of ε in ε-greedy is improved. In traditional ε-greedy strategies, an excessively large exploration rate will slow down or cause the algorithm to diverge, meaning the reward value cannot fluctuate within a certain range. Conversely, an excessively small exploration rate is not conducive to obtaining the optimal strategy. Considering that the agent can obtain more reward values ​​and converge within a finite time, the linearly decreasing ε value method used in most existing studies is changed to a curve-decreasing exploration rate. That is, in the later stages of iteration, more reliance is placed on empirical values, and the ε value is adjusted accordingly. t The design is to use the following value selection method:

[0068]

[0069] Where, ε initial ε is the initial exploration rate. final The exploration rate is the time to terminate, epoch is the current iteration number, and max_epoches is the maximum number of iterations. Figure 3 A comparison chart of parameter values ​​before and after the improvement of the ε-greedy strategy.

[0070] In this embodiment, the simulation test platform and environment are shown in Table 2. Table 3 shows the reinforcement learning simulation parameters and conditions for the unmanned mining locomotive.

[0071] Table 2 Simulation Test Platform and Environment

[0072]

[0073] Table 3. Reinforcement Learning Simulation Parameters and Conditions for Unmanned Mining Locomotives

[0074]

[0075] Figures 4 to 7 The results are respectively the simulation verification results of the present invention using actual specific parameters. Figure 4 The graph shows the distance-speed relationship after 500 iterations of the speed control method for unmanned electric locomotives used in mining. As the number of iterations increases, the vehicle speed curve gradually smooths out, and the vehicle can adjust its speed promptly when encountering obstacles, maintaining a set safe distance from them. Figure 5 The speed control method for unmanned electric locomotives in mining was iterated 500 times with distance-speed relationship and road condition map. It can be seen that as the number of iterations increases during the learning and iteration process, the vehicle can reach the maximum speed on straight roads and curves more quickly. Figure 6 A comparison of the reward values ​​obtained by the agent before and after the improvement of the ε-greedy policy is shown in the figure. It can be clearly seen from the figure that the reward value of the unmanned speed control of the mining locomotive based on the improved ε-greedy policy is higher and more convergent. Figure 7This chart compares the difference between the minimum following distance and the set standard following distance during the iterative process of the unmanned mining locomotive before and after the improvement of the ε-greedy strategy. The chart shows that the unmanned speed control of the mining locomotive using the improved ε-greedy strategy can better maintain a safe following distance.

[0076] All the above results demonstrate that this embodiment verifies that the speed control method for unmanned mining locomotives based on the ε-greedy strategy and reinforcement learning algorithm can effectively control the vehicle to run at the maximum set speed on different road sections and maintain a set safe distance from obstacles, thus realizing the unmanned autonomous operation of the mining locomotive. Furthermore, the improved ε-greedy strategy performs better in simulation applications and has higher agent learning efficiency.

[0077] The present invention and its embodiments have been described above illustratively. This description is not restrictive, and the figures shown are only one embodiment of the present invention. The actual structure and method are not limited thereto. Therefore, if those skilled in the art are inspired by this description and design similar structures and embodiments without departing from the spirit of the present invention, such designs shall fall within the protection scope of the present invention.

Claims

1. A speed control method for unmanned mining locomotives based on an improved ε-greedy strategy, characterized in that, Includes the following steps: (1) The mining unmanned electric locomotive is used as an intelligent agent, and the intelligent agent's actions are set to integers within the optimal driving / braking torque range applied to the axle of the locomotive. (2) Classify the operating status of intelligent agents based on the driving conditions of unmanned locomotives used in mining; (3) Design the agent reward function by setting different reward values ​​according to the agent's actions under different operating states; (4) Using the Q-Learning algorithm, the real-time status information of the locomotive is used as input, and the driving / braking torque applied to the axle and the reward value obtained by the agent are used as output to build a speed control model for the unmanned locomotive used in mining. (5) Use the ε-greedy policy to generate actions, and learn to update the Q function based on the agent's action at time t and the state and reward value obtained from the interaction with the environment at time t+1. (6) The reinforcement learning process is carried out within the set number of iterations. After reaching the maximum number of iterations, the speed control of the unmanned electric locomotive is completed.

2. The speed control method for unmanned mining locomotives according to claim 1, characterized in that, The driving status of the unmanned mining locomotive is divided into different states based on whether it travels on curved / straight roads, encounters obstacles, reaches its maximum speed, or reaches its destination.

3. The speed control method for unmanned mining locomotives according to claim 1 or 2, characterized in that, Different reward values ​​are set for the agent's actions in different states, and the reward function is set as follows: reward=c*(v record -v current ) Among them, v record It is the vehicle speed feedback value within the previous sampling interval, v current It is the vehicle speed feedback value at the current sampling time, and c is the reward function coefficient, the sign of which is determined by whether the vehicle is speeding in different states.

4. The speed control method for unmanned mining locomotives according to claim 1 or 2, characterized in that, The Q-function is learned and updated based on the agent's state and reward value at time t+1, obtained from the agent's interaction with the environment after the action taken at time t, as detailed below: Define the state S of the agent at time t. t and the action taken A t Define the Q function Q(S) t A t This is used to estimate the action value function q of all executable actions a in any state s when strategy π is adopted. π (s,a); Define the Q-value of an agent at time t+1 as Q(S) when the agent takes any action to interact with the environment at time t. t+1 ,a), using the method that makes Q(S t+1 a) Update Q(S) using the action with the largest value. t A t The updated formula is as follows: Q(S t ,A t )←Q(S t ,A t )+β[R t+1 +γmaxQ(S t+1 ,a)-Q(S t ,A t )] Where β is the learning rate, γ is the discount factor, and R0 t+1 Based on state S t And Action A t The reward value obtained.

5. The speed control method for unmanned mining locomotives according to claim 4, characterized in that, The action is generated using an ε-greedy strategy, the specific strategy of which is as follows: The agent's driving actions are randomly obtained according to the ε-greedy policy. The expression of the ε-greedy policy is as follows: Among them, a t Let ε be the action of the agent at time t, A be the set of all actions, and ε be the action of the agent at time t. t Let ε be the exploration rate of the agent at time t, i.e., the probability of randomly selecting an action, 0 < ε t <1, the probability of selecting the action amount based on the maximum Q function is 1-ε t ;ε t The method for obtaining the value is as follows: Where, ε initial ε is the initial exploration rate. final The exploration rate is the number of iterations at the termination time, epoch is the current iteration number, and max_epoches is the maximum number of iterations. Determine whether the difference between the action value to be taken at time t obtained according to the ε-greedy strategy and the action value at time t-1 exceeds a set threshold; If the threshold is exceeded, the ε-greedy strategy determines whether the vehicle should apply driving torque or braking torque at time t. Based on the action taken at time t-1, acceleration / deceleration is performed at time t. If the threshold is not exceeded, the action to be taken at time t determined by the ε-greedy strategy is executed.

Images