Heuristic mobile robot path planning method based on improved Q learning

Abstract

The invention provides a heuristic mobile robot path planning method based on improved Q learning, and belongs to the technical field of path planning. An obstacle map, a starting point and an ending point are given; transmitting the information to a trained neural network heuristic model, and outputting a heuristic line and a heuristic domain; enabling a heuristic domain output by the neural network heuristic model to initialize a Q table, and calculating a reward function of the scene by using an output heuristic line; the path planning of the agent is carried out through the QL algorithm iteration process, the agent inquires the current Q table to make a decision in each iteration, the reward value of the decision is calculated through the reward function, the Q table is updated, then the state of the agent after the decision is executed is obtained according to the state transfer function of the environment, and then the next iteration is carried out until the QL algorithm converges. According to the method, the problem of low generalization of a neural network heuristic model and the problem of reward sparseness of a reinforcement learning algorithm in solving a path planning task can be solved.

Description

TECHNICAL FIELD

[0001] The application relates to a heuristic mobile robot path planning method based on improved Q learning, and belongs to the technical field of path planning. BACKGROUND

[0002] Path planning is the basis for autonomous robots to move efficiently between different locations. An ideal path should be collision-free and enable the robot to reach the destination quickly with the shortest distance. Effective path planning not only requires avoiding obstacles, but also optimizes travel time and energy consumption to achieve efficient movement.

[0003] In recent years, artificial intelligence (AI) has become one of the indispensable core technologies for autonomous mobile robots. In the field of path planning, AI-based methods have gradually become an effective solution to overcome the limitations of traditional path planning methods, and related research has rapidly developed. Among these methods, Q learning (Q-Learning, hereinafter referred to as QL method) has shown broad application prospects in solving complex optimization problems due to its model independence and ability to learn and update knowledge through continuous interaction with the environment.

[0004] The core idea of Q learning is that the algorithm adjusts the decision-making according to the feedback (reward or punishment) of the state at each execution of an action. The goal of the QL method is to maximize the cumulative reward in a series of actions. To this end, QL uses a Q table (or lookup table) to record the expected return of each state-action combination, commonly referred to as Q value. Through repeated learning in the environment, Q values are constantly updated until an optimal action-state decision-making strategy is formed.

[0005] QL methods face a fundamental defect that the learning process becomes extremely slow in the case of high-dimensional state space and large-scale action space, mainly due to the "dimension curse" phenomenon. On the one hand, as the size of the environment or the action space increases, the increase in the dimension of the state space will cause the memory requirement of the Q table to grow exponentially, thereby limiting the application of QL methods in actual physical systems. On the other hand, traditional Q learning algorithms usually set the Q value to be equal or random in the initialization stage, i.e., the algorithm starts learning without prior knowledge, which leads to the problem of slow convergence speed. This limitation severely restricts the efficiency of QL in practical applications.

[0006] With the rapid development of artificial intelligence technology, a new perspective combining neural networks and path planning algorithms has gradually gained attention. In this emerging field, neural network models are trained to predict a potentially high-value region that serves as the sampling domain for a non-uniform sampler. Since this region contains the optimal path, samples are biased towards the optimal path, accelerating the convergence of the path planning algorithm.

[0007] Based on the background art, the present application proposes a new insight that the global prior heuristic information provided by the pre-trained model can effectively solve the slow convergence problem commonly encountered in Q-learning methods. Based on this insight, the present application further proposes an improved Q-learning method aimed at accelerating the convergence of the algorithm and optimizing its performance in complex environments by introducing a neural network-driven prediction mechanism. SUMMARY

[0008] The present application aims to solve the problems of the prior art and provide an improved Q-learning-based heuristic mobile robot path planning method.

[0009] The present application is achieved by the following technical solutions:

[0010] An improved Q-learning-based heuristic mobile robot path planning method, comprising the following steps:

[0011] Step 1: Given the obstacle map, starting point, and ending point;

[0012] Step 2: Input the information containing the obstacle map, starting point, and ending point in Step 1 into the trained neural network heuristic model, and output the heuristic line and heuristic domain after inference by the neural network heuristic model;

[0013] Step 3: Initialize the Q table using the heuristic domain output by the neural network heuristic model in Step 2, and then calculate the reward function of the scene using the output heuristic line;

[0014] Step 4: Perform path planning for the agent through the QL algorithm iteration process, use the initialized Q table obtained in Step 3, and at each iteration, the agent queries the current Q table to make a decision, calculates the reward value of the decision using the reward function obtained in Step 3, and updates the Q table, then obtains the agent state after executing the decision according to the state transition function of the environment, and then enters the next iteration until the QL algorithm converges.

[0015] Compared with the prior art, the present application has the following advantages:

[0016] 1. Low generalization problem of neural network heuristic model

[0017] The neural network heuristic model is given an obstacle map, a start point, and an end point. Through a large amount of training, the neural network can predict a high-value area containing an optimal path. It is undeniable that the higher the coincidence between the 'high-value area' and the real optimal path, the better the model learns the potential relationship between the obstacle map and the start and end points, and the better the 'high-value area' is for the heuristic of the reinforcement learning method. On the contrary, if the neural network heuristic model has poor prediction quality, it will greatly hinder the efficiency of the reinforcement learning for path planning, and even introduce false heuristic information.

[0018] Low generalization refers to the fact that although the neural network model can achieve good results in the test set, the model prediction effect is poor in a strange map that is greatly different from the training set map features. The low generalization of the neural network heuristic model is a technical problem to be solved by the present application. The dual-branch neural network heuristic model proposed by the present application effectively improves the generalization compared with other models in existing research, because it adopts two innovative designs: 1) A feature fusion module is introduced in the traditional encoder-decoder model architecture. When designing the feature fusion module, spatial attention and channel attention mechanisms are considered at the same time, effectively fusing low-level spatial details and high-level semantic information, improving the model's ability to learn the potential relationship between obstacles and start / end points, and thus improving the generalization; 2) Heuristic line-heuristic domain dual-branch supervised learning architecture. The traditional neural network heuristic model usually uses one kind of supervision signal, that is, the optimal path connecting the start point and the end point is obtained by a search algorithm in advance, and the optimal path (width of 1 pixel) is used as the true value to supervise the training of the model. The present application refers to the optimal path as a heuristic line, and widens the optimal path to a width of 20 pixels, called a heuristic domain. The strategy of using heuristic line-heuristic domain dual-signal supervised learning is used. Experiments show that this innovative design can also improve the model generalization. At the same time, the present application also proposes some auxiliary designs for the neural heuristic model to improve the model calculation efficiency. These designs include 1) using a STDC backbone network (Short-Term Dense Concatenate module) to construct an encoder, and 2) using a simplified pyramid pooling model.

[0019] 2. The reward sparsity problem existing in the reinforcement learning algorithm in solving the path planning task

[0020] In the path planning task, the reinforcement learning algorithm often faces the problem of sparse rewards, that is, the agent only gets significant rewards when reaching the final goal (such as reaching the end point), while most actions in the exploration process can only get zero or minimal feedback. This sparsity can lead to low training efficiency, convergence difficulties and other problems. At the same time, the agent is prone to local optimization (such as wandering in a safe area) or complete failure due to insufficient exploration, significantly slowing down the convergence speed or even causing the training to collapse. To alleviate this problem, researchers often inject dense learning signals by methods such as reward reshaping (adding distance guidance and other intermediate rewards), hierarchical strategy (decomposition into subtasks such as path finding / obstacle avoidance), or intrinsic motivation (encouraging exploration of unknown areas), in order to improve the efficiency and stability of path exploration. The key to solving the above-mentioned problem of sparse rewards lies in the proposed innovative reward function, which incorporates heuristic information from a neural network. Other methods for solving the problem of sparse rewards use a distance-guided strategy to construct a reward function, which fails to consider the relationship between scene obstacles and the starting point and the end point. Although this can improve the problem of sparse rewards, the effect is limited. Compared with the prior art, the present invention more effectively improves the problem of sparse rewards faced by traditional Q-learning methods and accelerates algorithm convergence, thanks to the following two innovative designs: 1) Introducing a reward function design based on neural network heuristic lines. Compared with distance-based reward functions, the heuristic information brought by neural network heuristic lines is more accurate, and this information is a global prior judgment of the environment by a pre-trained neural network, which is more robust than relying solely on distance as heuristic information, and can significantly reduce the occurrence of the algorithm falling into local optimization. 2) Introducing a Q-table initialization method based on neural network heuristic domains. Existing research shows that, compared to random initialization of Q-table, initialization methods with certain heuristic information can bring more efficiency improvement to Q-learning. The present invention uses neural network heuristic domain initialization, which can reduce the range of Q-learning algorithm iteration compared to distance-based initialization, and based on heuristic domain information, can avoid algorithm iteration in low-value areas from the beginning, greatly improving convergence speed. BRIEF DESCRIPTION OF DRAWINGS

[0021] Figure 1 The structure diagram of the neural network heuristic model of the present invention.

[0022] Figure 2 The advanced-low-level feature fusion diagram based on the attention mechanism of the present invention.

[0023] Figure 3 The structure diagram of the model heuristic domain and heuristic line prediction results of the present invention; wherein:

[0024] Figure 3 (a) is the structure diagram of the model heuristic domain and heuristic line prediction results of the present invention Figure 1 ;

[0025] Figure 3 (b) The structure of the model's heuristic domain and heuristic line prediction results of the present application Figure 2 ;

[0026] Figure 3 (c) The structure of the model's heuristic domain and heuristic line prediction results of the present application Figure 3 ;

[0027] Figure 3 (d) The structure of the model's heuristic domain and heuristic line prediction results of the present application Figure 4 ;

[0028] Figure 3 (e) The structure of the model's heuristic domain and heuristic line prediction results of the present application Figure 5 ;

[0029] Figure 3 (f) The structure of the model's heuristic domain and heuristic line prediction results of the present application

[0030] Figure 3 (g) The structure of the model's heuristic domain and heuristic line prediction results of the present application

[0031] Figure 3 (h) The structure of the model's heuristic domain and heuristic line prediction results of the present application

[0032] Figure 3 (i) The structure of the model's heuristic domain and heuristic line prediction results of the present application

[0033] Figure 4 The experimental comparison between the present application and the traditional method

[0034] Figure 4 (a) The four actions of the agent corresponding to each quadrant of the traditional method

[0035] Figure 4 (b) The agent's exploration area of the traditional method

[0036] Figure 4 (c) The agent's exploration area of the present application method

[0037] Figure 4 (d) The relationship between the number of iterations and rewards of the present application and the traditional method

[0038] Figure 5 The flowchart of the algorithm implementation of the present application DETAILED DESCRIPTION

[0039] The application will be further described in detail below with reference to the accompanying drawings: The present embodiment is implemented on the premise of the technical scheme of the application, and detailed implementation modes are given, but the protection scope of the application is not limited to the following examples.

[0040] As shown in Figures 1 to 5 , the heuristic mobile robot path planning method based on improved Q learning related by the present embodiment comprises:

[0041] 1. Double-branch neural network heuristic model

[0042] The input of the neural network heuristic model is the obstacle map and the start point and end point information, and the output is heuristic information , wherein the higher the value of a certain map grid, the more likely the model considers that the grid belongs to the optimal path.

[0043] The mainstream neural network prediction planning method adopts a three-channel RGB image as input, wherein the map is usually represented by black and white pixels, and the start point and end point are represented by blue and red pixels respectively. Unlike the above methods, the present application proposes the following four innovative model design ideas.

[0044] (1) The start point, end point and map are divided into three independent channels for input. STDC (Short-Term Dense Concatenate) is used as the backbone network instead of the traditional ResNet, and the model structure is as shown in the accompanying Figure 1 , the present application retains the first five feature extraction stages of the original STDC network, as shown in the accompanying Figure 1 , block_s2, block_s4, block_s8, block_s16, block_s32 represent the 1st to 5th stages respectively. In the multi-stage encoding architecture of the neural network heuristic model, the spatial resolution of the output feature tensor is reduced by half after each STDC; therefore, the STDC backbone network outputs two parts of content: the first part is the feature tensor output of the last stage, the 5th stage, which can extract the high-level semantic information of the input map, and is used as the input of the pyramid pooling module; the second part is the output of the 3rd, 4th and 5th stages, which is used for feature fusion in the decoder; the low-level features from the 3rd, 4th and 5th stages will be fused with the high-level features in the decoder in the next step to improve the performance of the model in the path planning task.

[0045] (2) Compared with the original pyramid pooling module (PPM for short), the application simplifies it to use three different pyramid scales and deletes the connection operation between the original feature mapping and the up-sampled features. The simplified PPM can reduce the calculation cost. The application uses three different pyramid scales. In detail, the feature tensor output of the STDC 5th stage is subjected to global average pooling with scales of 1x1, 2x2 and 4x4, respectively, to generate three layers of feature tensors with different scales, which are then subjected to convolution and up-sampling operations and connected together. Finally, a 1x1 convolution operation is used to obtain the final output of the pyramid pooling module. A smaller number of pyramid layers (such as 3 layers) can capture sufficient semantic information while reducing the calculation cost.

[0046] (3) The neural network inspired model uses a feature fusion module to fuse low-level spatial details (the outputs of the 3rd, 4th and 5th stages in the STDC) and high-level semantic information to improve the model generalization. This improvement not only improves the multi-scale understanding of the input data, but also enhances the reasoning ability of the model in complex scenarios, especially in application scenarios that require accurate path prediction. The feature fusion module design is shown in the accompanying Figure 2 , which includes a spatial attention branch and a channel attention branch. In detail, represents high-level semantic information, represents low-level spatial detail features, which are subjected to the following processes:

[0047]

[0048] In the above formula, is an axial average operation, is an axial maximum operation, represents a tensor concatenation operation along the axis. Let be a tensor with channel number C, width W and height H, i.e. , after taking the mean and maximum values along the channel axis, the dimension becomes , after connection, , then after convolution and sigmoid operation, the spatial attention weight is obtained; in the channel attention branch, the is subjected to average pooling and maximum pooling operations as follows:

[0049]

[0050] wherein, is an average pooling operation, is a maximum pooling operation, is a convolution operation, after average pooling and maximum pooling, the feature dimension becomes , after convolution and sigmoid operation, the channel attention weight is finally obtained Finally, the fused features output are calculated using the following formula. .

[0051]

[0052] (4) The neural network heuristic model has two independent decoders, which output a small-range 'heuristic line' and a large-range 'heuristic domain' respectively. Among them, the heuristic domain decoder contains multiple stages, as shown in the appendix. Figure 1 As shown, the features output by the pyramid pooling module and the features output from different stages in the STDC backbone network are fused together and then input into different stages of the heuristic domain decoder. The feature size in the decoder gradually increases from 8×8 to 32×32, while the channel dimension of the features gradually decreases from 512-dimensional to 128-dimensional. In fact, the neural network heuristic model is equivalent to a binary semantic segmentation task, so a conventional convolutional neural network layer is added after the heuristic domain decoder output to output the heuristic domain prediction result. For the heuristic line branch, the high-level-low-level feature fusion process is abandoned. The heuristic domain output is connected with the obstacle map and the start and end point information, and then used as the heuristic line output through a shallow U-Net structure decoder.

[0053] In actual training, classic path planning methods such as... Find the optimal path connecting the start and end points in a given map, and use this path as the heuristic line label. Expand the heuristic line outwards by a certain area as the heuristic domain label. During training, use the AdamW optimizer with an initial learning rate of 0.001 and a weight decay rate of 0.0001.

[0054] Experiments show that this dual-branch supervised learning strategy can improve the model's ability to learn the potential relationship between obstacles and the starting and ending points.

[0055] 2. An Improved Q-Learning Method for Path Planning Tasks

[0056] 2-1: Reward Prediction Method Based on Heuristic Lines

[0057] This invention proposes the following reward function:

[0058]

[0059] After performing action A, the agent reaches state S. Based on the relationship between state S and the environment, the reward value corresponding to the current step is calculated according to the above formula. If state S is an obstacle, it means the agent has encountered an obstacle, and the reward is -r1; if state S is a target point, it means the agent has reached the target, and the reward is r. max; if state S represents a position that has been visited, then a reward value of -r3 is given; and other position reward values of r for the remaining cases c In this embodiment, r1 = 10, r2 = 40, and r3 = 5. max

[0060] In the traditional Q-learning method, = 0, thus causing the reward sparsity problem, i.e., only when the agent explores the obstacle, the goal point, or the repeated position, a reward is given, which slows down the convergence speed of the algorithm.

[0061] The present application proposes a continuous reward function based on neural network prediction (referred to as N-CRF). This method uses the 'heuristic line' (denoted as ) generated by the neural network heuristic model to calculate the reward value associated with each state in the map. The specific calculation method is shown in the following formula.

[0062]

[0063] In the above formula, represents the reward value corresponding to state , and represents the value of the neural network heuristic line corresponding to state . In this embodiment, a two-dimensional grid scene is used, so state is represented by the plane coordinates .

[0064] The traditional method uses a distance-based reward function, as shown below,

[0065]

[0066] Meanwhile, this embodiment proposes to combine the distance-based reward function with the reward function based on neural network prediction, as shown in the following formula,

[0067]

[0068] wherein represents the basic reward for entering an empty space, and represents the maximum reward at the goal point. and respectively control the decay rate of the reward in the x-axis and y-axis directions. is the weight that adjusts the two reward functions, and respectively represent the x-axis and y-axis coordinates of the target position. The reward value r of other positions in the reward function is determined by the formula. c

[0069] ​​In the path planning based on reinforcement learning (RL), by designing a reward function, the agent obtains a higher reward when reaching the optimal region and a lower reward when deviating from the region, so that the agent can utilize prior knowledge instead of starting from scratch. Experimental results show that the method can accelerate the convergence process of Q learning.

[0070] 2-2: Q table initialization method based on heuristic domain

[0071] In the initial learning stage, the agent has limited understanding of the environment and usually selects actions randomly. In order to accelerate the convergence process of QL algorithm, it is proved that initializing Q table by prior knowledge is an effective method. Existing researches propose to use the Euclidean distance between the current state of the mobile robot and the target position to initialize the Q table. This method can provide certain guidance for the learning process and promote the algorithm to converge more quickly, thereby improving the overall performance.

[0072] The traditional method, i.e., the distance-based Q table initialization method (hereinafter referred to as D-QI), guides the agent towards the target by continuously prioritizing the action with the highest Q value at each state. However, this method only utilizes limited prior knowledge, and its initial Q value is not customized according to the specific environment. In the present invention, a new Q table initialization strategy is proposed, which utilizes the region prediction information generated by the neural network heuristic model, hereinafter referred to as N-QI, as shown in the following formula:

[0073]

[0074] wherein, represents the region prediction generated by the neural network heuristic model, and is a binary mask, where the region with high prediction value is assigned a value of 0, and the remaining regions are assigned a value of -10, as shown in the following formula. The weighting coefficient is used to adjust the balance between the two initialization methods.

[0075]

[0076] In the above formula, represents the prediction value of the position corresponding to the state in the region generated by the neural network heuristic model, when the prediction value is greater than a fixed threshold (thd), it is considered that this position is a high prediction value, and the value of this position is defined as 0, otherwise as -10. In the present specific embodiment, the fixed threshold thd is defined as 0.8.

[0077] The theoretical basis for using the region prediction generated by the neural network heuristic model as the Q table initialization condition is that the region prediction has stronger connectivity than the heuristic line prediction observed during the training process. Even if the heuristic line prediction of the neural network heuristic model achieves a higher score in F1 score, the lack of sufficient connectivity will still result in poor results when using these predictions as the Q table initialization condition. As shown in the unseen sample in Figure 3 Fig. 1, Figure 3 Fig. 2, Figure 3 (a), Figure 3 (b), Figure 3 (c), Figure 3 (d), Figure 3 (e), Figure 3 (f), and Figure 3 (g), the pink region is the heuristic domain predicted by the neural network model for this map, and the blue region is the heuristic line predicted by the neural network. The concentration of color represents the size of the value at a specific location in the heuristic domain, as shown in Figure 3 (f), the concentration of pink in some regions is low, representing the low value predicted by the neural network in that region. Figure 3 (f), Figure 3 (g), Figure 3 (h), and Figure 3 (i) are the same map scene and start and end points, Figure 3 (h), and Figure 3 (i) represent the true values of the heuristic line and the heuristic domain, respectively, Figure 3 (f), and Figure 3 (g) represent the visualization results of the heuristic domain and the heuristic line predicted by the neural network model, respectively. Since this map is an unseen scene, i.e., the obstacle arrangement is significantly different from the model training set, the heuristic line predicted by the neural network in Figure 3 (g) appears to be discontinuous, Figure 3 (f) the heuristic domain remains continuous.

[0078] The present application provides an improved Q-learning-based heuristic mobile robot path planning method, which uses a neural network heuristic model to solve the slow convergence problem of traditional QL methods applied in path planning tasks. The present application can significantly shorten the convergence time of the Q-learning method for path planning tasks.

[0079] The effectiveness of the present application is experimentally verified through the following specific experiments:

[0080] Experiment 1: Neural network heuristic model prediction effect

[0081] The proposed prediction model is validated using public datasets. The training and inference of all models are performed on an NVIDIA RTX 4090 GPU. To build the Q-learning experimental environment, the original map is scaled down to a smaller version.

[0082] The proposed neural network model is compared and evaluated with other state-of-the-art prediction models, including RGP, NEED, and MPT, focusing on the regional prediction accuracy on seen and unseen datasets.

[0083] According to the RGP model method, the prediction accuracy of the model is evaluated using the F1 score, which is between 0 and 1, and the higher the score, the higher the prediction accuracy of the model. The validation results are shown in the following table (Table 1). Compared with the previous state-of-the-art method RGP, the F1 score of the proposed neural network model on the seen dataset is improved by 5%, and on the unseen dataset is improved by 4%.

[0084] Table Performance comparison with existing methods, the values in the figure are F1 scores

[0085]

[0086] Experiment two, the effect of improved Q-learning path planning

[0087] The Q-table is visualized by dividing the map grid into four different quadrants along its diagonals, each corresponding to the four actions of the agent: forward, backward, left turn, and right turn, as shown in the accompanying Figure 4 Compared with the distance-based Q-table initialization method (traditional method), the proposed method effectively limits the interaction of the agent with specific regions in the environment by using the regional prediction output of the neural network heuristic model for initialization. Due to the influence of the greedy exploration strategy, the agent's exploration occasionally exceeds these designated regions. In addition, from the reward convergence trend in the training process, it can be seen that the traditional method converges after 1838 iterations, while the proposed method reduces this number to 1289, showing its higher efficiency and overall effect. Specific implementation one:

[0089] Before introducing the implementation, introduce related terms:

[0090] 1. State (state, S) The specific location of the system, in this invention, a grid map is used, at this time the position of each grid can be regarded as a state.

[0091] 2. Action, the action A that the agent can take in a particular state S, in this invention, the action includes moving up, down, left and right.

[0092] 3. Reward, the feedback that the agent gets when it moves from a state S1 to a state S2 by taking an action A, in this invention, the reward value is calculated using a reward function based on the next state S2.

[0093] 4. Q-Value, Q(S, A) represents the value of taking action A in state S, the higher the value, the more likely that action A will lead to a higher cumulative reward in the future.

[0094] 5. Learning rate, , determines the proportion of new information in each update of Q value.

[0095] 6. Reward decay factor γ, represents the current value of future rewards.

[0096] The specific embodiments of the present invention are described in conjunction with the accompanying Figure 5 The specific embodiments of the present invention are described in conjunction with the accompanying

[0097] Step 1: Given the obstacle map and the start and end points, this invention does not discuss how the obstacles and start and end points are obtained, assuming they are known, and the map is given in the form of a two-dimensional grid, for example, the map is 256x256 resolution, each white grid (rgb value is 255, 255, 255) represents a free passage area, black grid (rgb is 0, 0, 0) represents an obstacle, the start point uses a red (255, 0, 0) grid, and a blue grid (0, 0, 255) represents the end point.

[0098] Step 2: The map containing obstacles and start and end points is fed into the trained neural network heuristic model, and after model inference, the heuristic line and heuristic domain are output, both of which have a size of 256x256, and the value of each grid is from 0 to 1, the closer the value is to 1, the greater the probability that the model considers that the grid contains the optimal path.

[0099] Step 3: Use the heuristic line output by the neural network heuristic model to initialize the Q table, then use the heuristic domain to calculate the reward function of the scene;

[0100] Step 4: Start the QL algorithm iteration process, in each iteration, the agent queries the current Q table to make a decision, calculates the reward value of the decision through the reward function, and updates the Q table, then gets the state of the agent after executing the decision according to the state transition function of the environment, then enters the next iteration until the algorithm converges.

[0101] The detailed process in step 4 is as follows:

[0102] Step 4-1: Find an action A according to the current state S in the Q-value table. In the first iteration, use the starting point as the current state. Use the ε-greedy strategy, that is, randomly select an action from the four actions with probability ε, and find the current Q-value table with probability 1-ε, select the action with the highest Q-value corresponding to the current state S.

[0103] Step 4-2: Perform action A, move from the previous state S1 to the next state S2, and use the reward function proposed in the invention to calculate the reward value R according to the current state S2.

[0104] Step 4-3: Update the Q-value table using the Bellman equation as follows

[0105]

[0106] The left side of the above equation is the new Q-value of action A in state S, and the right side is the original Q-value, represents the next state all actions The maximum Q-value in the next state.

[0107] Step 4-4 Complete one state transition, return to step 4-1, and continue until the algorithm converges. Convergence conditions of the algorithm: reach the maximum number of iterations, or reach the target state.

[0108] Terminology explanation:

[0109] Q-learning: A model-free reinforcement learning algorithm based on value function, its core idea is to approximate the optimal policy by iteratively updating the action value function.

[0110] Pyramid pooling module of neural network: Pyramid pooling module is a deep learning structure for multi-scale context information aggregation, its core idea is to capture global and local context information of images through different scale pooling operations, and enhance the robustness of the model to object size changes.

[0111] Encoder and decoder of neural network: Encoder and decoder is a symmetrical structure, widely used in data compression, feature extraction and generation tasks (such as image segmentation, machine translation, generative adversarial network, etc.). The encoder compresses the input data (such as images, text) into high-level latent representation. The decoder reconstructs or generates the target data (such as restoring images, translating text) from the latent representation.

[0112] The above merely describes preferred specific embodiments of the present application, which are based on different implementations of the overall concept of the present application, and the protection scope of the present application is not limited thereto. Any changes or replacements that are easily conceived by those skilled in the art within the technical scope disclosed by the present application shall be covered by the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A heuristic path planning method for mobile robots based on improved Q-learning, characterized in that, Includes the following steps: Step 1: Given an obstacle map, a starting point, and an ending point; Step 2: Input the information including the obstacle map, starting point and ending point from Step 1 into the trained neural network heuristic model. After the neural network heuristic model infers, it outputs heuristic lines and heuristic domains. Step 3: Initialize the Q-table using the heuristic domain output by the neural network heuristic model in Step 2. Then, use the output heuristic lines to calculate the reward function of the scene. Step 4: Perform path planning for the agent through the iterative process of the QL algorithm. Using the initial Q-table obtained in Step 3, in each iteration, the agent queries the current Q-table to make a decision, calculates the reward value of the decision through the reward function obtained in Step 3, and updates the Q-table. Then, the agent's state after executing the decision is obtained according to the state transition function of the environment, and then the next iteration begins, until the QL algorithm converges.

2. The heuristic mobile robot path planning method based on improved Q-learning according to claim 1, characterized in that, The specific structure of the neural network heuristic model described in step two is as follows: (1) The neural network heuristic model uses STDC as the backbone network, which contains 5 feature extraction stages. In the multi-stage coding architecture of the neural network heuristic model, the spatial resolution of the output feature tensor will be reduced by half after each STDC stage. The STDC backbone network outputs two parts: the first part is the feature tensor output of the 5th stage of the 5 feature extraction stages, which can extract the high-level semantic information of the input map and is used as the input of the pyramid pooling module; the second part is the output of the 3rd, 4th and 5th stages of the 5 feature extraction stages, which is used in the decoder as low-level spatial details for feature fusion. (2) The neural network heuristic model adopts a simplified pyramid pooling module and uses three different pyramid scales to process the feature tensor output of the STDC stage 5 through global average pooling at three different pyramid scales to generate feature tensors of three different scales. Then, convolution and upsampling operations are performed and the tensors are connected together. Finally, convolution operation is used to obtain the final output of the pyramid pooling module. (3) The neural network-inspired model uses feature fusion to combine low-level spatial details and high-level semantic information. The feature fusion module includes spatial attention branches and channel attention branches. Represents high-level semantic information. Representing low-level spatial detail features, spatial attention features are obtained through the following process. : （1） In the above formula, For axial averaging operation. This is for the operation of the maximum axial value. Represents a tensor splicing operation along the axis; let... Let C be a tensor with C channels, W width, and H height, i.e. After taking the mean and maximum values ​​along the channel axis, the dimension becomes After connection, Then, through convolution and sigmoid operations, spatial attention weights are obtained. ; In the channel attention branch, for Perform average pooling and max pooling operations as follows: （2） in, For average pooling operation, For max pooling operation, For convolution operations, after average pooling and max pooling, the feature dimensions become After convolution and sigmoid operations, the channel attention weights are finally obtained. Finally, the fused features output are calculated using the following formula. ; （3） (4) The neural network heuristic model has two independent decoders, which output a small range of heuristic lines and a large range of heuristic domains respectively. For the heuristic domain branch, the heuristic domain decoder contains multiple stages. The features output by the simplified pyramid pooling module and the features output by different stages in the STDC backbone network are input to different stages of the heuristic domain decoder after passing through the feature fusion module. The feature size in the heuristic domain decoder gradually increases, while the channel dimension of the features gradually decreases. A conventional convolutional neural network layer is added after the output of the heuristic domain decoder to output the heuristic domain prediction result. For the heuristic line branch, the heuristic domain output is connected with the obstacle map, the starting point and the ending point information, and the heuristic line is output through the decoder of the shallow U-Net structure.

3. The heuristic mobile robot path planning method based on improved Q-learning according to claim 1, characterized in that, The specific method for initializing the Q-table using the heuristic domain output by the neural network heuristic model in step two, as described in step three, is as follows: A novel Q-table initialization strategy is proposed, which utilizes region prediction information generated by a neural network heuristic model, hereinafter referred to as N-QI, as shown in the following equation: （4） in, This represents the region prediction generated by a neural network-inspired model, while It is a binary mask where regions with high predicted values ​​are assigned a value of 0, and the remaining regions are assigned a value of -10, as shown in the following formula; weighting coefficients. Used to balance the two initialization methods; （5） In the above formula, This indicates that in the region generated by the neural network-inspired model, the state is... The predicted value corresponding to a given location is defined as follows: if the predicted value is greater than a fixed threshold thd, then this location is considered a high predicted value, and a threshold value is defined for that location. The value is 0, otherwise it is -10.

4. The heuristic mobile robot path planning method based on improved Q-learning according to claim 3, characterized in that, The specific method for calculating the scene's reward function using the output heuristic line described in step three is as follows: The following reward function is proposed: （6） After performing action A, the agent reaches state S. Based on the relationship between state S and the environment, the reward value corresponding to the current step is calculated according to the above formula (6). If state S is an obstacle, it means that the agent has encountered an obstacle, and the reward is -r1. If state S is a target point, it means that the agent has reached the target, and the reward is r. max If state S represents a previously visited position, then the reward value is -r3; for other positions, the reward value is r. c ; Then, a continuous reward function based on neural network prediction is proposed, utilizing the heuristic line generated by the neural network heuristic model, denoted as... Calculate the reward value associated with each state in the map. The specific calculation method is shown in the following formula: （7） In the above formula, Representing state The reward value corresponding to other position states in formula (6) is given. Representing state The value of the corresponding neural network heuristic line at that location; Traditional methods use a distance-based reward function, as shown in Equation (8). （8） This paper proposes to combine a distance-based reward function with a continuous reward function based on neural network prediction, as shown in the following formula (9): （9） in, This represents the basic reward for entering an open space, while This indicates the maximum reward at the target point. and Control the decay rate of the reward in the x-axis and y-axis directions respectively. It involves adjusting the weights of the two reward functions. and These represent the x-axis and y-axis coordinates of the target location, respectively.

5. The heuristic mobile robot path planning method based on improved Q-learning according to claim 4, characterized in that, The specific steps for path planning of the agent through the QL algorithm iteration process described in step four are as follows: Step 41: Based on the current state S, look up the Q-value table and select an action A. In the first iteration, use the starting point as the current state. Use the ε-greedy strategy, that is, randomly select one action from the four actions with a probability of ε, and look up the current Q-value table with a probability of 1-ε to select the action with the highest Q-value corresponding to the current state S. Step 42: Execute action A, transition from the previous state S1 to the next state S2, use the output heuristic domain to calculate the reward function of the scene, and calculate the reward value R based on the current state S2; Step 43: Update the Q-value table using the Bellman equation, as follows: （10） Left side of the above equation It is the new Q value of action A in state S, on the right. It is the original Q value. Indicate the next state All actions The largest Q value; Step 44: Complete one state transition, return to Step 41, and repeat until the algorithm converges. The convergence condition of the algorithm is reaching the maximum number of iterations or reaching the target state.

Images