Multi-locomotive path planning method and system based on Q-learning algorithm

Abstract

The invention discloses a multi-locomotive path planning method and system based on a Q-learning algorithm, and relates to the technical field of iron-steel interface locomotive path planning, and the method comprises the steps: obtaining environment data, building a transportation system environment model, and defining states and actions; performing path optimization on the vehicle path through a Q-learning algorithm to obtain an initial path plan; and the central control system receives the key information in real time, and dynamically adjusts vehicle path planning in real time through a Q-learning algorithm. According to the method, through learning and optimization of the Q-learning algorithm, the central control system can find the shortest path, the transportation time and distance are reduced, and the transportation efficiency is improved; the path planning is dynamically adjusted through the central control system, so that the congestion and conflicts of track routes are avoided, and the smoothness of the transportation process is ensured; by using the Q-learning algorithm, the method can adapt to a dynamically changing environment, and normal operation of the system can be ensured even if an accident occurs.

Description

Technical Field

[0001] The present invention relates to the technical field of locomotive path planning at an iron-steel interface, and in particular to a multi-locomotive path planning method and system based on a Q-learning algorithm. Background Art

[0002] The digital transformation and upgrading of the steel industry has become an inevitable trend. In the process of pursuing efficient and intelligent production, modern steel companies are increasingly relying on advanced information technology and automation technology. As a key link in steel production, the efficiency and safety of molten iron transportation directly affect the overall production efficiency and cost control. Traditional molten iron transportation path planning methods are often difficult to meet actual needs in the face of complex and changing environments.

[0003] Traditional path planning methods, such as the A* algorithm and the Dijkstra algorithm, rely on accurate environmental models and deterministic environmental information. These algorithms work efficiently in static and fully known environments, but their performance drops significantly in dynamic and uncertain environments. The molten iron transportation system in a steel plant is often dynamically changing due to equipment failures, path blockages, and other reasons, which makes it difficult for traditional algorithms to respond and adjust the path in real time. In addition, when faced with large-scale and complex rail networks, traditional algorithms have high computational complexity and long path planning time, making it difficult to meet the needs of rapid decision-making. When dealing with multi-tank car coordination problems, traditional path planning algorithms are prone to path conflicts and usually require the introduction of complex scheduling strategies, which increases the complexity and computational burden of the system.

[0004] Modern steel companies need an efficient and reliable path planning method that can cope with complex dynamic environments to improve the overall efficiency and safety of molten iron transportation. Traditional blind search algorithms perform global searches by traversing all possible paths. Although they can find solutions, they are extremely inefficient. Heuristic search algorithms can more effectively avoid obstacles and find the optimal path by introducing specific optimization rules. However, even heuristic search does not perform well in highly dynamic and uncertain industrial environments.

[0005] In this context, the reinforcement learning algorithm based on Q-learning provides a new solution for molten iron transportation path planning. As a model-free reinforcement learning algorithm, Q-learning has significant advantages in learning the optimal strategy through continuous interaction with the environment. First, the Q-learning algorithm does not rely on an accurate environmental model, but adapts to environmental changes through real-time feedback and updates, which enables it to maintain a high path planning efficiency in a dynamic and uncertain environment. Secondly, Q-learning can gradually optimize the path selection strategy through iteration and continuous updating of the Q table, and its path planning effect will continue to improve over time. Even if the initial path selection is not optimized enough, it can eventually approach the optimal path through multiple iterations and adjustments. In addition, the Q-learning algorithm can find a balance point in multi-objective optimization and improve overall performance by designing a reasonable reward function and comprehensively considering multiple optimization objectives, such as the shortest path, minimum energy consumption, and avoiding conflicts.

[0006] Therefore, the application of Q-learning-based reinforcement learning algorithm in molten iron transportation path planning can significantly improve transportation efficiency and system responsiveness. This not only meets the needs of modern steel enterprises for efficient production, but also provides strong technical support for the development of intelligent manufacturing and industrial automation. With the continuous advancement of technology, the application prospects of Q-learning algorithm in molten iron transportation system will be broader, bringing new impetus to the digital transformation and upgrading of the steel industry. Summary of the invention

[0007] In view of the above-mentioned problems, the present invention is proposed.

[0008] Therefore, the technical problems solved by the present invention are: the existing technology relies on precise models, deterministic environmental information, and takes a long time to plan paths in complex rail networks; and how to solve the problem of extremely low efficiency of existing algorithms in complex dynamic environments.

[0009] To solve the above technical problems, the present invention provides the following technical solutions: a multi-locomotive path planning method based on a Q-learning algorithm, comprising acquiring environmental data, establishing a transportation system environment model, and defining states and actions; optimizing the vehicle path through a Q-learning algorithm to obtain an initial path planning; a central control system receives key information in real time and dynamically adjusts the vehicle path planning in real time through a Q-learning algorithm.

[0010] As a preferred solution of the multi-locomotive path planning method based on the Q-learning algorithm described in the present invention, the environmental data includes a track network, signal lights, switches, and vehicle positions; the track network is modeled as a directed graph, in which nodes represent track positions and edges represent connections between tracks; each node and edge has corresponding attributes, and the corresponding attributes include track length, signal light status, and switch status.

[0011] As a preferred solution of the multi-locomotive path planning method based on the Q-learning algorithm described in the present invention, the state and action include the operating state of the vehicle in the transportation system and the action performed; the state includes the current position of the vehicle, the target position, the occupancy of the track, the state of the signal light and the switch; the action includes the operation of moving the vehicle from the current track to the adjacent track.

[0012] As a preferred solution of the multi-locomotive path planning method based on the Q-learning algorithm described in the present invention, wherein: the path optimization of the vehicle path by the Q-learning algorithm includes initializing a Q value for each state and action, the Q value can be set to 0 or a random value, and the dimension of the Q table is expressed as:

[0013] S|×|A

[0014] Among them, |S| represents the number of states, and |A| represents the number of actions. After initialization, the vehicle will select an action through the ε-greedy strategy according to the current state at each time step and execute it. The ε-greedy strategy randomly selects an action with a probability of ε and selects the action with the largest current Q value with a probability of 1-ε. The formula of the ε-greedy strategy is expressed as:

[0015]

[0016] Among them, a t represents the action selected at time step t, s t Represents the state at time step t; after executing the action, the vehicle will enter a new state and obtain immediate rewards based on the environmental feedback results, and update the Q value; update the Q value table, and the updated Q value formula is expressed as:

[0017] Q(s t ,a t )←Q(s t ,a t )+α[r t +γmaxQ(s t+1 ,a')-Q(s t ,a t )]

[0018] Among them, r trepresents the immediate reward obtained at time step t, s t+1 represents the new state at time step t+1, α represents the learning rate, γ represents the discount factor, and a' represents the estimated action; the value range of α is 0<α≤1, and the value range of γ is 0≤γ<1.

[0019] As a preferred solution of the multi-locomotive path planning method based on the Q-learning algorithm described in the present invention, the instant reward includes a reward obtained based on a reward function judgment criterion when the vehicle enters a new state and according to environmental feedback; the reward function includes path length, transportation efficiency, safety, and energy consumption; the path length is the path length from the starting point to the end point of the vehicle, and the shorter the path, the higher the reward. Assuming that the path length is d, the reward function is expressed as:

[0020] r=-d

[0021] Transportation efficiency refers to whether the vehicle can complete the transportation task on time. If the task is completed on time, the reward is high, and if it is delayed, the reward is deducted. Assume that the reward for completing the task on time is r on_time , the penalty for delay is r delay , then the reward function is expressed as:

[0022]

[0023] Safety means that the vehicle should avoid collisions with other vehicles during driving. If a collision occurs, a heavy penalty will be imposed. Assume that the penalty for a collision is r conflict , then the reward function is expressed as:

[0024] r=-r conflict

[0025] Energy consumption is to choose a path with low energy consumption. The reward for a low energy consumption path is high. Assuming the energy consumption is e, the reward function is expressed as:

[0026] r=-e

[0027] Among them, r represents the immediate reward.

[0028] As a preferred solution of the multi-locomotive path planning method based on the Q-learning algorithm described in the present invention, the initial path planning includes planning an initial driving route for each vehicle according to the current Q table; the initial path is to select a path with the highest reward according to the existing Q value in the Q table, and the calculation formula of the initial path is expressed as:

[0029]

[0030] Among them, s represents the state and a represents the action.

[0031] As a preferred solution of the multi-locomotive path planning method based on the Q-learning algorithm described in the present invention, wherein: the real-time dynamic adjustment of vehicle path planning through the Q-learning algorithm includes: during the actual operation process, the vehicle collects vehicle status and environmental status data in real time through GPS and sensors, and transmits the collected data to the central control system in real time through the wireless network. The central control system calculates the current optimal path of the vehicle based on the acquired data by running the Q-learning algorithm, and then the central control system will feed back the calculated optimal path to the vehicle in real time through the wireless network, and the vehicle will travel according to the path recommended by the central control system; and during the driving process, the vehicle will also feed back the actual operation status to the central control system in real time, which is used to further optimize the Q table and achieve real-time planning of the optimized path closed loop; the path calculation formula of the central control system for calculating the optimal path based on real-time data is expressed as:

[0032]

[0033] Among them, t represents the current time and T represents the total time.

[0034] Another object of the present invention is to provide a multi-locomotive path planning system based on the Q-learning algorithm, which can run the Q-learning algorithm through a central control system and calculate the optimal path based on real-time data, thereby solving the problem that the prior art is difficult to respond and adjust the planned path in real time due to dynamic environmental changes.

[0035] As a preferred solution of the multi-locomotive path planning system based on the Q-learning algorithm described in the present invention, it includes:

[0036] A computer device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement a multi-locomotive path planning method based on a Q-learning algorithm.

[0037] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a multi-locomotive path planning method based on a Q-learning algorithm.

[0038] Beneficial effects of the present invention: The multi-locomotive path planning method based on the Q-learning algorithm provided by the present invention can find the shortest path through the learning and optimization of the Q-learning algorithm, and the central control system can find the shortest path, reduce the transportation time and distance, and improve the transportation efficiency; through the dynamic adjustment path planning of the central control system, the track route congestion and conflict are avoided, and the smooth transportation process is ensured; the optimal path can be planned through the algorithm prediction of the system, the conflict between vehicles is avoided, and the safety during the transportation process is ensured. At the same time, the system can also monitor the status of the vehicle and the track conditions in real time, and timely discover and deal with potential safety hazards; the central control system calculates through the Q-learning algorithm, and the system can select the path with the lowest energy consumption, which not only reduces energy consumption, but also reduces operating costs; by using the Q-learning algorithm, the present invention can adapt to the dynamically changing environment, even if unexpected situations occur, it can also ensure the normal operation of the system, and at the same time the system performs path planning based on real-time data and algorithms, realizes data-driven decision-making, and improves the level of intelligent manufacturing. The present invention achieves better results in dynamic environment processing, real-time path planning and algorithm efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0040] Figure 1 An overall flow chart of a multi-locomotive path planning method based on a Q-learning algorithm provided for the first embodiment of the present invention.

[0041] Figure 2 An overall detailed design diagram of a multi-locomotive path planning method based on a Q-learning algorithm provided for the first embodiment of the present invention.

[0042] Figure 3 A tank truck path driving diagram of a multi-locomotive path planning method based on a Q-learning algorithm is provided in the second embodiment of the present invention.

[0043] Figure 4 An overall flow chart of a multi-locomotive path planning system based on a Q-learning algorithm provided in the third embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the above-mentioned purposes, features and advantages of the present invention more obvious and easy to understand, the specific implementation methods of the present invention are described in detail below in conjunction with the drawings of the specification. Obviously, the described embodiments are part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary persons in the art without creative work should fall within the scope of protection of the present invention.

[0045] Example 1, reference Figure 1-Figure 2 , as an embodiment of the present invention, provides a multi-locomotive path planning method based on Q-learning algorithm, comprising:

[0046] S1: Obtain environmental data, establish a transportation system environment model, and define states and actions.

[0047] Furthermore, environmental data includes data on the track network, signal lights, switches, vehicle locations, etc.

[0048] It should be noted that the state and action include the operating state of the vehicle and the actions performed in the transportation system.

[0049] It should also be noted that the track network is modeled as a directed graph, in which nodes represent track positions and edges represent connections between tracks; each node and edge has corresponding attributes, including track length, signal light status, and switch status.

[0050] It should also be noted that the state includes the current position of the vehicle, the target position, the occupancy of the track, the status of the signal lights and switches; the action includes the operation of moving the vehicle from the current track to the adjacent track.

[0051] S2: Optimize the vehicle path through the Q-learning algorithm to obtain the initial path planning.

[0052] Furthermore, the Q-learning algorithm is used to optimize the vehicle path, including initializing a Q value for each state and action. The Q value can be set to 0 or a random value. The dimension of the Q table is expressed as:

[0053] S|×|A

[0054] Among them, |S| represents the number of states, and |A| represents the number of actions. After initialization, the vehicle will select an action through the ε-greedy strategy according to the current state at each time step and execute it. The ε-greedy strategy randomly selects an action with a probability of ε and selects the action with the largest current Q value with a probability of 1-ε. The formula of the ε-greedy strategy is expressed as:

[0055]

[0056] Among them, a t represents the action selected at time step t, s t Represents the state at time step t; after executing the action, the vehicle will enter a new state and obtain immediate rewards based on the environmental feedback results, and update the Q value; update the Q value table, and the updated Q value formula is expressed as:

[0057] Q(s t ,a t )←Q(s t ,a t )+α[r t +γmaxQ(s t+1 ,a')-Q(s t ,a t )]

[0058] Among them, r t represents the immediate reward obtained at time step t, s t+1 represents the new state at time step t+1, α represents the learning rate, γ represents the discount factor, and a' represents the estimated action; the value range of α is 0<α≤1, and the value range of γ is 0≤γ<1.

[0059] It should be noted that the immediate reward includes the reward obtained when the vehicle enters a new state and based on the environmental feedback based on the reward function judgment criteria; the reward function includes path length, transportation efficiency, safety, and energy consumption; the path length is the path length from the starting point to the end point of the vehicle, and the shorter the path, the higher the reward. Assuming the path length is d, the reward function is expressed as:

[0060] r=-d

[0061] Transportation efficiency refers to whether the vehicle can complete the transportation task on time. If the task is completed on time, the reward is high, and if it is delayed, the reward is deducted. Assume that the reward for completing the task on time is r on_time , the penalty for delay is r delay , then the reward function is expressed as:

[0062]

[0063] Safety means that the vehicle should avoid collisions with other vehicles during driving. If a collision occurs, a heavy penalty will be imposed. Assume that the penalty for a collision is r conflict , then the reward function is expressed as:

[0064] r=-r conflict

[0065] Energy consumption is to choose a path with low energy consumption. The reward for a low energy consumption path is high. Assuming the energy consumption is e, the reward function is expressed as:

[0066] r=-e

[0067] Among them, r represents the immediate reward.

[0068] It should also be noted that the initial path planning includes planning the initial driving route for each vehicle according to the current Q table; the initial path is to select a path with the highest reward according to the existing Q value in the Q table, and the calculation formula of the initial path is expressed as:

[0069]

[0070] Among them, s represents the state and a represents the action.

[0071] S3: The central control system receives key information in real time and dynamically adjusts the vehicle path planning in real time through the Q-learning algorithm.

[0072] Furthermore, the real-time dynamic adjustment of vehicle path planning through the Q-learning algorithm includes that during the actual operation, the vehicle collects vehicle status and environmental status data in real time through GPS and sensors, and transmits the collected data to the central control system in real time through the wireless network. The central control system calculates the current optimal path of the vehicle based on the acquired data by running the Q-learning algorithm. The central control system then feeds back the calculated optimal path to the vehicle in real time through the wireless network, and the vehicle will travel according to the path recommended by the central control system.

[0073] It should be noted that during driving, the vehicle will also feed back the actual operating conditions to the central control system in real time, which will be used to further optimize the Q table and achieve real-time planning and optimization of the path closed loop.

[0074] It should also be noted that the path calculation formula for the central control system to calculate the optimal path based on real-time data is expressed as:

[0075]

[0076] Among them, t represents the current time and T represents the total time.

[0077] It should also be said that key information includes the real-time position of the vehicle, track status, signal lights and switch status light data.

[0078] Example 2, reference Figure 3 , which is an embodiment of the present invention, provides a multi-locomotive path planning method based on Q-learning algorithm. In order to verify the beneficial effects of the present invention, scientific demonstration is carried out through economic benefit calculation and simulation experiments.

[0079] The present invention evaluates the effect of the path optimization method based on the Q-learning algorithm by testing and verifying an actual molten iron transportation system.

[0080] A track network model was established in the molten iron transportation system of a steel plant, and relevant data was collected and entered, including the topological structure of the track network, the status of signal lights and switches, the initial position of the tank car, etc.

[0081] By simulating a variety of transportation tasks, training the Q-learning algorithm, and updating the Q value in the Q table, it can effectively respond to path planning needs in different environments. According to the trained Q table, the optimal path is planned for each tank truck, and the optimal path is selected based on the current track status and tank truck position.

[0082] The tank truck travels along the recommended route and provides real-time feedback of the actual operation status to the system. During the actual operation, the system receives the tank truck position and track status information in real time, dynamically adjusts the route planning, and ensures the smooth completion of the transportation task. If the track fails or a new tank truck is added, the system can update the Q table in time and re-plan the route.

[0083] At the same time, the tank truck continuously provides feedback on track occupancy, signal lights, switch status and other information during its driving process. The system adjusts the Q table based on this feedback data to improve the accuracy and adaptability of path planning.

[0084] Through these steps, the present invention demonstrates the effectiveness and superiority of the molten iron transportation path optimization method based on the Q-learning algorithm in a dynamic environment.

[0085] In actual application, three tank cars are randomly selected and set off from different starting points to reach the designated location. In the experimental scenario, three tank cars set off from different starting points to the designated location. Car No. 1 goes from the No. 1 tap hole of Blast Furnace I to the No. 1 tap hole of Blast Furnace II, Car No. 2 goes from the No. 8 tap hole of Blast Furnace I to the No. 7 tap hole of Blast Furnace II, and Car No. 3 goes from the No. 6 tap hole of Blast Furnace II to Casting Line 2. The driving path diagram of the three tank cars from the starting point to the target location is shown in the figure below. Figure 3 The experimental results are visualized through SVG files, showing the tanker's driving trajectory. The tanker's trajectory is filled with three colors: orange, red, and pink. The white light indicates the tanker's expected driving section, and the signal light status is also marked accordingly.

[0086] from Figure 3It can be seen from the trajectory process diagram shown that the No. 3 tank car represented by the orange trajectory line starts from the No. 6 tap hole of the No. II blast furnace and chooses the shortest path as its optimal route to travel through the 35 / 42G track and safely arrive at the target Casting Line 2; the No. 1 tank car represented by the orange trajectory line starts from the No. 1 tap hole of the No. I blast furnace. In order to avoid the No. 3 tank car and avoid a collision, it chooses the line where the 15 / 23G track is located as its optimal path to travel and successfully arrives at the No. 1 tap hole of the No. II blast furnace; and the No. 2 tank car represented by the red trajectory line chooses to "stop and wait" at the 4 / 14G track to reduce energy consumption as its optimal path, and waits until the No. 3 tank car passes and safely arrives at the No. 7 tap hole of the No. II blast furnace by the shortest path.

[0087] From this, it can be seen that the multi-locomotive path planning method based on the Q-learning algorithm provided by the present invention finds a balance point in multi-objective optimization through reasonable reward functions, such as the shortest path, the lowest energy consumption, avoiding conflicts, etc., and achieves an improvement in the overall performance. At the same time, it can also be seen that the present invention can efficiently and reliably plan the optimal path when facing a complex dynamic environment, thereby improving the overall efficiency and safety of molten iron transportation.

[0088] Example 3, reference Figure 4 , as an embodiment of the present invention, provides a multi-locomotive path planning system based on Q-learning algorithm, including a state action definition module, an initial path planning module, and a real-time path processing module.

[0089] The state-action definition module is used to obtain environmental data, establish a transportation system environmental model, and define states and actions; the initial path planning module is used to optimize the vehicle path through the Q-learning algorithm to obtain the initial path planning; the real-time path processing module is used for the central control system to receive key information in real time and dynamically adjust the vehicle path planning in real time through the Q-learning algorithm.

[0090] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0091] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0092] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0093] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, multiple steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc. It should be noted that the above embodiments are only used to illustrate the technical solution of the present invention and are not limited. Although the present invention is described in detail with reference to the preferred embodiments, it should be understood by those skilled in the art that the technical solution of the present invention can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of the present invention, which should be included in the scope of the claims of the present invention.

[0094] It should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solutions of the present invention may be modified or replaced by equivalents without departing from the spirit and scope of the technical solutions of the present invention, which should all be included in the scope of the claims of the present invention.

Claims

1. A multi-locomotive path planning method based on Q-learning algorithm, characterized in that: include: Acquire environmental data, build a transportation system environment model, and define states and actions; The vehicle path is optimized through the Q-learning algorithm to obtain the initial path planning; The central control system receives key information in real time and dynamically adjusts vehicle path planning in real time through the Q-learning algorithm.

2. The multi-vehicle path planning method based on Q-learning algorithm as claimed in claim 1, characterized in that: The environmental data includes track network, signal lights, switches, and vehicle positions; The track network is modeled as a directed graph, where nodes represent track locations and edges represent connections between tracks; Each node and edge has corresponding attributes, including track length, signal light status, and switch status.

3. The multi-vehicle path planning method based on Q-learning algorithm as claimed in claim 2, characterized in that: The states and actions include the operating states of vehicles in the transportation system and the actions performed; The status includes the current position of the vehicle, the target position, the occupancy of the track, and the status of the signal lights and switches; An action includes the operation of moving a vehicle from the current track to an adjacent track.

4. The multi-vehicle path planning method based on Q-learning algorithm as claimed in claim 3, characterized in that: The path optimization of the vehicle path by the Q-learning algorithm includes initializing a Q value for each state and action. The Q value can be set to 0 or a random value. The dimension of the Q table is expressed as: S|×|A Among them, |S| represents the number of states, |A| represents the number of actions; After initialization, at each time step, the vehicle will select an action based on the current state through the ε-greedy strategy and execute it; The ε-greedy strategy is to randomly select an action with a probability of ε, and select the action with the largest current Q value with a probability of 1-ε. The formula of the ε-greedy strategy is expressed as: Among them, a t represents the action selected at time step t, s t represents the state at time step t; After executing the action, the vehicle will enter a new state and obtain immediate rewards based on the environmental feedback results, and update the Q value; Update the Q value table, and the updated Q value formula is expressed as: Q(s t ,a t )←Q(s t ,a t )+α[r t +γmaxQ(s t+1 ,a')-Q(s t ,a t )] Among them, r t represents the immediate reward obtained at time step t, s t+1 represents the new state at time step t+1, α represents the learning rate, γ represents the discount factor, and a' represents the estimated action; The value range of α is 0<α≤1, and the value range of γ is 0≤γ<1.

5. The multi-vehicle path planning method based on Q-learning algorithm as claimed in claim 4, characterized in that: The instant reward includes a reward obtained based on a reward function judgment criterion when the vehicle enters a new state and according to environmental feedback; The reward function includes path length, transportation efficiency, safety, and energy consumption; The path length is the length of the path from the starting point to the end point of the vehicle. The shorter the path, the higher the reward. Assuming the path length is d, the reward function is expressed as: r＝-d Transportation efficiency refers to whether the vehicle can complete the transportation task on time. If the task is completed on time, the reward is high, and if it is delayed, the reward is deducted. Assume that the reward for completing the task on time is r on_time , the penalty for delay is r delay , then the reward function is expressed as: Safety means that the vehicle should avoid collisions with other vehicles during driving. If a collision occurs, a heavy penalty will be imposed. Assume that the penalty for a collision is r conflict , then the reward function is expressed as: r＝-r conflict Energy consumption is to choose a path with low energy consumption. The reward for a low energy consumption path is high. Assuming the energy consumption is e, the reward function is expressed as: r＝-e Among them, r represents the immediate reward.

6. The multi-vehicle path planning method based on Q-learning algorithm as claimed in claim 5, characterized in that: The initial path planning includes planning an initial driving route for each vehicle according to the current Q table; The initial path is to select a path with the highest reward based on the existing Q value in the Q table. The calculation formula of the initial path is expressed as: Among them, s represents the state and a represents the action.

7. The multi-vehicle path planning method based on Q-learning algorithm as claimed in claim 6, characterized in that: The real-time dynamic adjustment of vehicle path planning by the Q-learning algorithm includes, during the actual operation, the vehicle collects vehicle status and environmental status data in real time through GPS and sensors, and transmits the collected data to the central control system in real time through the wireless network. The central control system calculates the current optimal path of the vehicle based on the acquired data by running the Q-learning algorithm, and then the central control system feeds back the calculated optimal path to the vehicle in real time through the wireless network, and the vehicle will travel according to the path recommended by the central control system; During the driving process, the vehicle will also provide real-time feedback of the actual operation status to the central control system for further optimization of the Q table and achieve real-time planning and optimization of the closed loop path. The path calculation formula for the central control system to calculate the optimal path based on real-time data is expressed as: Among them, t represents the current time and T represents the total time.

8. A system using the multi-motorcycle path planning method based on the Q-learning algorithm as claimed in any one of claims 1 to 7, characterized in that: It includes state action definition module, initial path planning module and real-time path processing module; The state action definition module is used to obtain environmental data, establish a transportation system environment model, and define states and actions; The initial path planning module is used to optimize the vehicle path through the Q-learning algorithm to obtain the initial path planning; The real-time path processing module is used for the central control system to receive key information in real time and dynamically adjust the vehicle path planning in real time through the Q-learning algorithm.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the multi-motorcycle path planning method based on the Q-learning algorithm described in any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-locomotive path planning method based on the Q-learning algorithm described in any one of claims 1 to 7 are implemented.

Images