Path planning method and device, equipment and medium

Abstract

The invention provides a path planning method and device, equipment and a medium, and relates to the technical field of path planning. In the application, firstly, an initial reward matrix is constructed based on a distribution relationship among a plurality of region nodes in a target region; secondly, based on the initial reward matrix, initial accumulated reward value distribution is determined; then, based on the initial accumulated reward value distribution, determining at least two region nodes in the plurality of region nodes, so as to construct a target planning path of the target moving object; furthermore, after the target moving object moves based on the target planning path, the initial reward matrix is updated based on the moving state of the target moving object in the moving process, an updated reward matrix is formed, and updated accumulated reward value distribution is determined based on the updated reward matrix. On the basis of the content, the problem that in the prior art, the reliability of path planning is relatively low can be solved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of path planning, in particular to a path planning method, device, equipment and medium. BACKGROUND

[0002] Automated Guided Vehicle (AGV) is widely used in intelligent workshops to realize automatic and efficient transportation, and the AGV path planning problem brought by this has become a research hotspot. Among them, path planning can be divided into traditional algorithms and intelligent algorithms. Traditional algorithms include A* algorithm, Dijkstra algorithm, Floyd algorithm, artificial potential field method, etc. The traditional algorithm is prone to problems such as difficulty in environment modeling, slow convergence speed of algorithm, and easy to fall into local optimal solution. Intelligent algorithms include genetic algorithm, particle swarm optimization algorithm, neural network algorithm, reinforcement learning algorithm, etc. Genetic algorithm and particle swarm optimization algorithm are prone to fall into local optimum, neural network algorithm needs a large number of sample data for learning and training, and the training time is long, and reinforcement learning algorithm obtains knowledge through the exploration of agents in the environment, and learns in trial and error, which is widely used in path planning problems.

[0003] For intelligent workshops, there are many factors affecting AGV operation in the production site, and the production environment is complex and changeable, so the reliability of path planning is relatively low. SUMMARY

[0004] Therefore, the purpose of the present application is to provide a path planning method, device, equipment and medium to improve the problem of relatively low reliability of path planning in the prior art.

[0005] To achieve the above purpose, the technical scheme adopted by the present application is as follows: A path planning method comprises: Based on the distribution relationship between a plurality of region nodes in a target region, an initial reward matrix is constructed, wherein each reward parameter in the initial reward matrix is used to reflect the reward size obtained from the movement from a corresponding first region node to a second region node; Based on the initial reward matrix, an initial cumulative reward value distribution is determined, wherein each cumulative reward value in the initial cumulative reward value distribution is used to reflect the cumulative value of the reward obtained from the movement from the corresponding first region node to the second region node and the reward obtained from the movement from the second region node to other region nodes; Based on the initial cumulative reward value distribution, at least two region nodes are determined in the plurality of region nodes to construct a target planning path of a target motion object; After the target moving object moves based on the target planned path, the initial reward matrix is ​​updated based on the movement state of the target moving object during the movement process to form an updated reward matrix, and the updated cumulative reward value distribution is determined based on the updated reward matrix, wherein the updated cumulative reward value distribution serves as the basis for path planning for the target moving object or other moving objects.

[0006] In a preferred embodiment of this application, in the above path planning method, the steps of updating the initial reward matrix based on the motion state of the target moving object during the movement process after the target moving object moves according to the target planned path to form an updated reward matrix, and determining the updated cumulative reward value distribution based on the updated reward matrix, include: After the target moving object moves based on the target planned path, based on the movement state of the target moving object during the movement process, the congestion characterization parameters between every two adjacent regional nodes in the target planned path are determined; Based on the congestion characterization parameters between every two adjacent regional nodes in the target planning path, the reward parameters between the corresponding regional nodes in the initial reward matrix are updated to form an updated reward matrix, and the updated cumulative reward value distribution is determined based on the updated reward matrix.

[0007] In a preferred embodiment of this application, in the above path planning method, the step of determining the congestion characterization parameter between every two adjacent regional nodes in the target planned path based on the motion state of the target moving object during the movement process after the target moving object moves according to the target planned path includes: After the target moving object moves based on the target planned path, for every two adjacent regional nodes in the target planned path, the movement speed and path congestion duration of the target moving object between the two adjacent regional nodes are obtained; Based on the movement speed and the path congestion duration, congestion characterization parameters between the two adjacent regional nodes are determined. Specifically, based on the current updated cumulative reward value distribution, at least two regional nodes are identified from among the multiple regional nodes to construct a new target planning path. After the corresponding moving object moves based on the new target planning path, during the update of the current update reward matrix, the determined congestion characterization parameters are the weighted sum of the currently determined congestion characterization parameters and each historically determined congestion characterization parameter. The weighting coefficient decreases as the number of times the moving object moves, representing the reduction in the influence of previous movement conditions on the current situation.

[0008] In a preferred embodiment of this application, in the above path planning method, the step of updating the reward parameters between corresponding regional nodes in the initial reward matrix based on the congestion characterization parameters between every two adjacent regional nodes in the target planned path to form an updated reward matrix, and determining the updated cumulative reward value distribution based on the updated reward matrix, includes: For every two adjacent regional nodes in the target planning path, the difference between the reward parameters between the two adjacent regional nodes and the congestion characterization parameters between the two adjacent regional nodes in the initial reward matrix is ​​calculated to obtain the updated reward parameters, thereby forming the updated reward matrix; The distribution of the updated cumulative reward value is determined based on the updated reward matrix.

[0009] In a preferred embodiment of this application, the step of constructing an initial reward matrix based on the distribution relationship between multiple regional nodes in the target region in the above path planning method includes: Obtain the distance between any two regional nodes in the target region; Based on the distance between every two region nodes, construct the adjacency matrix of the target region; An initial reward matrix is ​​constructed based on the adjacency matrix. For each pair of connected regional nodes in the target region, a reward parameter between the two regional nodes is determined based on the maximum value in the adjacency matrix and the adjacency parameter between the two regional nodes. For each pair of unconnected regional nodes in the target region, the reward parameter between the two regional nodes is configured as a target value, wherein the target value is less than the reward parameter between any two connected regional nodes.

[0010] In a preferred embodiment of this application, the step of determining the initial cumulative reward value distribution based on the initial reward matrix in the above path planning method includes: For every two regional nodes in the target region, the reward parameter obtained from moving from the first regional node to the second regional node is obtained from the initial reward matrix to obtain the first reward parameter, and the maximum value of the reward parameter obtained from moving from the second regional node to each other regional node is obtained to obtain the second reward parameter. Based on the first reward parameter, the second reward parameter, and the discount factor configured for the second reward parameter, the cumulative reward value between the two regional nodes is determined; The initial cumulative reward value distribution is obtained based on the cumulative reward value between every two regional nodes in the target region.

[0011] In a preferred embodiment of this application, in the above path planning method, the step of determining at least two regional nodes from the plurality of regional nodes based on the initial cumulative reward value distribution to construct the target planning path for the target moving object includes: The starting region node and the destination region node of the target moving object are determined from the plurality of region nodes; Using the starting region node as the starting node and the destination region node as the ending node, based on the initial cumulative reward value distribution, the system searches backwards for the node with the maximum reward that has not been visited before, in order to form the target planning path for the target moving object.

[0012] This application also provides a path planning device, comprising: The reward matrix construction module is used to construct an initial reward matrix based on the distribution relationship between multiple regional nodes in the target area. Each reward parameter in the initial reward matrix is ​​used to reflect the reward obtained from moving from the corresponding first regional node to the second regional node. The reward value determination module is used to determine the initial cumulative reward value distribution based on the initial reward matrix, wherein each cumulative reward value in the initial cumulative reward value distribution is used to reflect the cumulative value of the reward obtained from moving from the corresponding first regional node to the second regional node and the reward obtained from moving from the second regional node to other regional nodes; The path planning module is used to determine at least two regional nodes among the multiple regional nodes based on the initial cumulative reward value distribution, so as to construct the target planning path of the target moving object; The reward value update module is used to update the initial reward matrix based on the motion state of the target moving object during the motion process after the target moving object moves according to the target planned path, forming an updated reward matrix, and to determine the updated cumulative reward value distribution based on the updated reward matrix, wherein the updated cumulative reward value distribution serves as the basis for path planning for the target moving object or other moving objects.

[0013] Based on the above, this application also provides an electronic device, including: Memory, used to store computer programs; A processor connected to the memory is used to execute the computer program stored in the memory to implement the path planning method described above.

[0014] Based on the above, this application also provides a computer-readable storage medium storing a computer program that executes the various steps of the path planning method described above when the computer program is run.

[0015] The path planning method, apparatus, device, and medium provided in this application first construct an initial reward matrix based on the distribution relationship between multiple regional nodes in a target area; second, based on the initial reward matrix, determine the initial cumulative reward value distribution; then, based on the initial cumulative reward value distribution, determine at least two regional nodes among the multiple regional nodes to construct a target planning path for the target moving object; further, after the target moving object moves according to the target planning path, update the initial reward matrix based on the movement state of the target moving object during the movement process to form an updated reward matrix, and determine the updated cumulative reward value distribution based on the updated reward matrix. Based on the above, since the update is based on the movement state of the target moving object during the movement process, prior knowledge can be fully utilized, thereby ensuring a more reliable planning basis in subsequent path planning. Therefore, the reliability of path planning can be improved, thus addressing the problem of relatively low reliability in existing path planning technologies. Attached Figure Description

[0016] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, preferred embodiments are described below in detail with reference to the accompanying drawings.

[0017] Figure 1 A structural block diagram of an electronic device provided in an embodiment of this application.

[0018] Figure 2 This is a flowchart illustrating the path planning method provided in an embodiment of this application.

[0019] Figure 3 The working map provided for the embodiments of this application requires path planning.

[0020] Figure 4 This is a schematic diagram of the first result of path planning provided in an embodiment of this application.

[0021] Figure 5 This is a schematic diagram of a second result of path planning provided in an embodiment of this application.

[0022] Figure 6 This is a schematic diagram illustrating a third result of path planning provided in an embodiment of this application.

[0023] Figure 7 This is a block diagram of a path planning device provided in an embodiment of this application. Detailed Implementation

[0024] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations.

[0025] Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely to illustrate selected embodiments of the application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0026] like Figure 1 As shown in the illustration, this application provides an electronic device. The electronic device may include a memory, a processor, and a path planning device.

[0027] Specifically, the memory and the processor are electrically connected directly or indirectly to enable data transmission or interaction. For example, the memory and the processor can be electrically connected via one or more communication buses or signal lines. The path planning device includes at least one software functional module stored in the memory in the form of software or firmware. The processor is used to execute executable computer programs stored in the memory, such as the software functional modules and computer programs included in the path planning device, to implement the path planning method provided in the embodiments of this application.

[0028] Optionally, the memory may be, but is not limited to, random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), etc.

[0029] Furthermore, the processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), a system on chip (SoC), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0030] Understandable. Figure 1 The structure shown is for illustrative purposes only; the electronic device may also include components that are more advanced than those shown. Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown may include, for example, a communication unit for exchanging information with other devices.

[0031] Combination Figure 2 This application also provides a path planning method applicable to the aforementioned electronic device. The method steps defined in the process of the path planning method can be implemented by the electronic device. The following will describe... Figure 2 The specific process shown will be explained in detail.

[0032] Step S110: Construct an initial reward matrix based on the distribution relationship between multiple regional nodes in the target region.

[0033] In this embodiment, the electronic device can construct an initial reward matrix based on the distribution relationship between multiple regional nodes in the target area. Each reward parameter in the initial reward matrix reflects the reward obtained from moving from the corresponding first regional node to the second regional node; that is, each reward parameter corresponds to two regional nodes.

[0034] Step S120: Determine the initial cumulative reward value distribution based on the initial reward matrix.

[0035] In this embodiment of the application, after constructing the initial reward matrix, the electronic device can determine the initial cumulative reward value distribution based on the initial reward matrix. Each cumulative reward value in the initial cumulative reward value distribution reflects the cumulative value of the reward obtained from moving from the corresponding first regional node to the second regional node (i.e., the reward for the current action) and the reward obtained from moving from the second regional node to other regional nodes (i.e., attention to subsequent actions).

[0036] Step S130: Based on the initial cumulative reward value distribution, at least two regional nodes are determined among the multiple regional nodes to construct the target planning path of the target moving object.

[0037] In this embodiment, after determining the initial cumulative reward value distribution, the electronic device can determine at least two region nodes from the plurality of region nodes based on the initial cumulative reward value distribution to construct a target planning path for the moving object. For example, path planning can be performed along the direction with a larger cumulative reward value.

[0038] Step S140: After the target moving object moves based on the target planned path, the initial reward matrix is ​​updated based on the movement state of the target moving object during the movement process to form an updated reward matrix, and the updated cumulative reward value distribution is determined based on the updated reward matrix.

[0039] In this embodiment, after constructing the target planned path and after the target moving object moves based on the target planned path, the electronic device can update the initial reward matrix based on the motion state of the target moving object during the movement process (i.e., prior knowledge) to form an updated reward matrix, and determine the updated cumulative reward value distribution based on the updated reward matrix. The updated cumulative reward value distribution serves as the basis for path planning for the target moving object or other moving objects.

[0040] Based on the above, since the motion state of the target moving object is updated accordingly during the motion process, prior knowledge can be fully utilized, thereby ensuring a more reliable planning basis in subsequent path planning. Therefore, the reliability of path planning can be improved, thus addressing the problem of relatively low reliability in existing path planning technologies.

[0041] Firstly, regarding step S110, it should be noted that the specific method for constructing the initial reward matrix is ​​not restricted and can be selected according to actual needs.

[0042] For example, in an alternative implementation, in order to ensure that the constructed initial reward matrix has high reliability, the above step S110 may further include steps S111, S112 and S113, the specific contents of each step are as follows.

[0043] Step S111: Obtain the distance between every two regional nodes in the target area.

[0044] In this embodiment of the application, the distance between any two regional nodes in a target area can be obtained. It should be noted that the distance can refer to the length of the road between the two regional nodes. If there is no road between two regional nodes, i.e. they are not connected, the distance between the two regional nodes is configured to a large value, such as infinity or much larger than the distance between any two connected regional nodes.

[0045] Step S112: Construct the adjacency matrix of the target region based on the distance between every two region nodes.

[0046] In this embodiment, after obtaining the distance between any two region nodes, an adjacency matrix of the target region can be constructed based on the distance between any two region nodes. For example, when the number of the plurality of region nodes is n, the size of the adjacency matrix can be n*n.

[0047] Step S113: Construct an initial reward matrix based on the adjacency matrix.

[0048] In this embodiment, after constructing the adjacency matrix of the target region, an initial reward matrix can be constructed based on the adjacency matrix. For every two connected region nodes in the target region, a reward parameter between the two region nodes is determined based on the maximum value in the adjacency matrix and the adjacency parameter between the two region nodes. For every two unconnected region nodes in the target region, the reward parameter between the two region nodes is configured as a target value. The target value is less than the reward parameter between any two connected region nodes.

[0049] For example, the initial reward matrix can be obtained based on the following formula: (1) in, For matrix The largest value in the adjacency matrix is ​​the farthest distance between nodes in the region. It is a positive coefficient, when it is passable (connected) and When it is not the finish line, ,when When it is the finish line, This means that you will receive a higher reward at the finish line; Defined according to the actual situation, it is a relatively large value, representing... Impassable (not connected), walk The path will be penalized more severely. Therefore, taking the shorter path will yield a greater reward.

[0050] Secondly, regarding step S120, it should be noted that the specific method for determining the distribution of the initial cumulative reward value is not restricted and can be selected according to actual needs.

[0051] For example, in an alternative implementation, in order to ensure the reliability of the determined initial cumulative reward value distribution, the above step S120 may further include steps S121, S122 and S123, the specific contents of each step are as follows.

[0052] Step S121: For every two regional nodes in the target region, obtain the reward parameter obtained from moving from the first regional node to the second regional node from the initial reward matrix to obtain the first reward parameter, and obtain the maximum value of the reward parameter obtained from moving from the second regional node to each other regional node to obtain the second reward parameter.

[0053] In this embodiment of the application, for every two regional nodes in the target region, the reward parameter obtained from moving from the first regional node to the second regional node is obtained from the initial reward matrix to obtain the first reward parameter, and the maximum value among the reward parameters obtained from moving from the second regional node to each other regional node is obtained to obtain the second reward parameter.

[0054] Step S122: Based on the first reward parameter, the second reward parameter, and the discount factor configured for the second reward parameter, determine the cumulative reward value between the two regional nodes.

[0055] In this embodiment, after obtaining the first reward parameter and the second reward parameter, the cumulative reward value between the two regional nodes can be determined based on the first reward parameter, the second reward parameter, and a discount factor configured for the second reward parameter. The discount factor can be used to characterize the degree of attention to future action rewards; the higher the degree of attention, the larger the value of the discount factor, and vice versa.

[0056] For example, the cumulative reward value can be determined using the following formula: (2) That is, at the regional node The selected action is to move to the region node. Driving, this action earns a reward. (i.e., the first reward parameter), at the regional node You can choose to travel to different next nodes, and select the one with the highest reward parameter for updating. Additionally, it should be noted that... The update is an iterative process; the number of iterations can be set, and iteration stops if the number of iterations is sufficient or the algorithm has converged. Additionally, , is a discount factor, representing foresight (i.e., the degree of attention paid to future actions). This means that only the reward of the current action is considered.

[0057] Step S123: Based on the cumulative reward value between every two regional nodes in the target region, obtain the initial cumulative reward value distribution.

[0058] In this embodiment of the application, after obtaining the cumulative reward value between every two regional nodes, an initial cumulative reward value distribution can be obtained based on the cumulative reward value between every two regional nodes in the target region. For example, the size of the initial cumulative reward value distribution is n*n, where n is the number of regional nodes.

[0059] Thirdly, regarding step S130, it should be noted that the specific method for constructing the target planning path of the target moving object is not restricted and can be selected according to actual needs.

[0060] For example, in an alternative implementation, in order to ensure the reliability of the constructed target planning path, the starting area node and the destination area node of the target moving object can be determined from the plurality of area nodes; then, with the starting area node as the starting node and the destination area node as the ending node, based on the initial cumulative reward value distribution, the node with the maximum reward that has not been passed can be searched one by one to form the target planning path of the target moving object.

[0061] For example, in another alternative implementation, in order to ensure the reliability of the constructed target planning path, the above step S130 may further include steps S131, S132 and S133, the specific contents of each step are as follows.

[0062] Step S131: Determine the starting region node and the destination region node of the target moving object from the multiple region nodes, and traverse the multiple region nodes multiple times to form multiple candidate planning paths.

[0063] In this embodiment, the starting and destination region nodes of the target moving object are determined from the plurality of region nodes, and the plurality of region nodes are traversed multiple times to form multiple candidate planning paths (it should be noted that all paths can be traversed, or only some paths can be traversed). In each candidate planning path, the first region node is the starting region node, and the last region node is the destination region node. That is, the target moving object needs to move from the starting region node to the destination region node.

[0064] Step S132: For each candidate planning path, determine the sum of the cumulative reward values ​​of every two adjacent regional nodes in the initial cumulative reward value distribution.

[0065] In this embodiment of the application, after determining the plurality of candidate planning paths, for each candidate planning path, the sum of the cumulative reward values ​​of every two adjacent regional nodes in the candidate planning path in the initial cumulative reward value distribution can be determined, such as the cumulative reward value between the first regional node and the second regional node + the cumulative reward value between the second regional node and the third regional node + ... + the cumulative reward value between the second-to-last regional node and the last regional node.

[0066] Step S133: The candidate planning path whose sum of cumulative reward values ​​is the minimum among the multiple candidate planning paths is determined as the target planning path of the target motion object.

[0067] In this embodiment of the application, after obtaining the sum of the cumulative reward values ​​of each candidate planning path, the candidate planning path whose sum of the cumulative reward values ​​is the maximum value can be determined as the target planning path of the target moving object.

[0068] Fourthly, regarding step S140, it should be noted that the specific method for determining the distribution of the updated cumulative reward value is not restricted and can be selected according to actual needs.

[0069] For example, in an alternative implementation, in order to make full use of prior knowledge, that is, to fully consider the motion state of the target moving object during the motion process, so that the reliability of the resulting updated cumulative reward value distribution is high, the above step S140 can further include steps S141 and S142, the specific contents of each step are as follows.

[0070] Step S141: After the target moving object moves based on the target planned path, based on the movement state of the target moving object during the movement process, determine the congestion characterization parameters between every two adjacent regional nodes in the target planned path.

[0071] In this embodiment of the application, after the target moving object moves based on the target planned path, a congestion characterization parameter between every two adjacent regional nodes in the target planned path can be determined based on the movement state of the target moving object during the movement process. The congestion characterization parameter can be used to characterize the congestion situation of the target moving object on the road between the corresponding two regional nodes. Thus, the larger the congestion characterization parameter, the more severe the congestion.

[0072] Step S142: Based on the congestion characterization parameters between every two adjacent regional nodes in the target planning path, update the reward parameters between the corresponding regional nodes in the initial reward matrix to form an updated reward matrix, and determine the updated cumulative reward value distribution based on the updated reward matrix.

[0073] In this embodiment, after determining the congestion characterization parameters, the reward parameters between corresponding regional nodes in the initial reward matrix can be updated based on the congestion characterization parameters between every two adjacent regional nodes in the target planning path, forming an updated reward matrix. Furthermore, the updated cumulative reward value distribution is determined based on the updated reward matrix. In other words, the determined updated cumulative reward value distribution carries relevant information about the motion state of the target moving object during its movement, i.e., prior knowledge.

[0074] It is understood that in step S141 above, the specific method of determining the congestion characterization parameters between every two adjacent regional nodes in the target planning path is not limited. For example, in an alternative implementation, in order to fully characterize the congestion situation between two adjacent regional nodes through the determined congestion characterization parameters, step S141 above may further include steps S141a and S141b, the specific contents of each step are as follows.

[0075] Step S141a: After the target moving object moves based on the target planned path, for every two adjacent regional nodes in the target planned path, obtain the movement speed and path congestion duration of the target moving object between the two adjacent regional nodes.

[0076] In this embodiment of the application, after the target moving object moves based on the target planned path, for every two adjacent regional nodes in the target planned path, the movement speed and path congestion duration of the target moving object between the two adjacent regional nodes can be obtained, that is, the movement speed and path congestion duration (i.e., movement duration) in this movement. Step S141b: Based on the movement speed and the path congestion duration, determine the congestion characterization parameters between the two adjacent regional nodes.

[0077] In this embodiment, after obtaining the movement speed and the path congestion duration, a congestion characterization parameter between two adjacent regional nodes can be determined based on the movement speed and the path congestion duration. For example, the movement speed and the path congestion duration can be multiplied to obtain the corresponding congestion characterization parameter. Specifically, based on the current updated cumulative reward value distribution, at least two regional nodes are identified from the plurality of regional nodes to construct a new target planning path. After the corresponding moving object moves based on the new target planning path, during the process of updating the current updated reward matrix, the determined congestion characterization parameter is the weighted sum of the currently determined congestion characterization parameter and each historically determined congestion characterization parameter. The weighting coefficient decreases as the number of times the moving object moves increases, representing the decrease in the influence of previous movement conditions on the current situation. For example, the congestion characterization parameter can be characterized by the following formula: ; in, To count backwards from the current motion, the total number of times each moving object has moved is counted. This represents the path congestion duration during the nth movement. This represents the velocity during the nth movement. This is the attenuation coefficient. For example, This represents the congestion time during the previous path traversal. If no congestion occurred, then... , This represents the duration of congestion during the second passage through this path prior to this movement. Because... The presence of a congestion means that the impact of a single path blockage will decrease as the number of moves increases.

[0078] It is understood that the specific method for determining the updated cumulative reward value distribution in step S142 above is not limited. For example, in an alternative implementation, in order to ensure the reliability of the determined updated cumulative reward value distribution, step S142 above may further include steps S142a and S142b, the specific contents of each step are as follows.

[0079] Step S142a: For every two adjacent regional nodes in the target planning path, calculate the difference between the reward parameters between the two adjacent regional nodes and the congestion characterization parameters between the two adjacent regional nodes in the initial reward matrix to obtain updated reward parameters, thereby forming an updated reward matrix.

[0080] In this embodiment of the application, for every two adjacent regional nodes in the target planning path, the difference between the reward parameter between the two adjacent regional nodes and the congestion characterization parameter between the two adjacent regional nodes in the initial reward matrix can be calculated to obtain the updated reward parameter (e.g., equal to the reward parameter minus the congestion characterization parameter) to form the updated reward matrix.

[0081] Step S142b: Determine the distribution of the updated cumulative reward value based on the updated reward matrix.

[0082] In this embodiment of the application, after obtaining the updated reward matrix, the updated cumulative reward value distribution can be determined based on the updated reward matrix. The specific processing procedure can be referred to the explanation of step S120 above, that is, the initial cumulative reward value distribution is determined based on the initial reward matrix.

[0083] To facilitate understanding of the path planning method described above, this application embodiment also provides a specific application example, which can be referred to in detail. Figures 3-6 .

[0084] Step 1: Build a working map by Figure 3 For example, this will solve the path planning problem between two points. ).in, Figure 3 This is an undirected graph, meaning the AGV's running direction is not restricted. The numbers on the lines represent the distance between two points, and its corresponding adjacency matrix is:

[0085] Step 2: Calculate the initial route First, the adjacency matrix can be processed according to the aforementioned formula (1) to obtain the corresponding initial reward matrix, where can be set Obtain the reward matrix for:

[0086] Then, the Q table can be initialized as follows: A 0 matrix, set Update the Q-table (i.e., the initial cumulative reward value distribution) according to the above formula (2), set the maximum number of iterations to 100, and obtain the final route result, as shown in the figure. Figure 4As shown.

[0087] Step 3: Plan the path based on prior knowledge To verify the effectiveness of the optimized algorithm, we assume... The path segment was congested for 10 minutes for some reason. Assuming the AGV's running speed is 1, the reward matrix is ​​updated as follows:

[0088] The new path planning result obtained at this time is as follows: Figure 5 As shown, the AGV successfully avoided... Path. If , Both paths are congested, and the path planning results are as follows: Figure 6 As shown, this verifies the effectiveness of the path planning method described above, successfully avoiding high-risk paths and significantly reducing AGV waiting time in production practice, thereby improving production efficiency.

[0089] It should be noted that, in Figures 4-6 In the diagram, the red line segment represents the target planning path.

[0090] Combination Figure 7 This application also provides a path planning device applicable to the aforementioned electronic device. The path planning device may include a reward matrix construction module, a reward value determination module, a planned path construction module, and a reward value update module.

[0091] The reward matrix construction module can be used to construct an initial reward matrix based on the distribution relationship between multiple regional nodes in the target area. Each reward parameter in the initial reward matrix reflects the reward obtained from moving from the corresponding first regional node to the second regional node. In this embodiment, the reward matrix construction module can be used to execute... Figure 2 The details of step S110, and the relevant content regarding the reward matrix construction module, can be found in the preceding description of step S110.

[0092] The reward value determination module can be used to determine an initial cumulative reward value distribution based on the initial reward matrix, wherein each cumulative reward value in the initial cumulative reward value distribution reflects the cumulative value of the reward obtained from moving from the corresponding first regional node to the second regional node and the reward obtained from moving from the second regional node to other regional nodes. In this embodiment of the application, the reward value determination module can be used to execute... Figure 2 The details of step S120, and the relevant content regarding the reward value determination module, can be found in the preceding description of step S120.

[0093] The path planning module can be used to determine at least two regional nodes from the plurality of regional nodes based on the initial cumulative reward value distribution, so as to construct a target planning path for the target moving object. In this embodiment, the path planning module can be used to execute... Figure 2 The relevant content regarding the planned path construction module in step S130 shown can be found in the previous description of step S130.

[0094] The reward value update module can be used to update the initial reward matrix based on the motion state of the target moving object during its movement after the target moving object has moved according to the target planned path, forming an updated reward matrix, and to determine the updated cumulative reward value distribution based on the updated reward matrix, wherein the updated cumulative reward value distribution serves as the basis for path planning for the target moving object or other moving objects. In this embodiment, the reward value update module can be used to perform... Figure 2 The details of step S140, and the relevant content regarding the reward value update module, can be found in the preceding description of step S140.

[0095] In this application embodiment, corresponding to the path planning method applied to the electronic device described above, a computer-readable storage medium is also provided, which stores a computer program that executes the various steps of the path planning method when the computer program is run.

[0096] The steps executed by the aforementioned computer program during runtime will not be described in detail here; please refer to the explanation of the path planning method above.

[0097] In summary, the path planning method, apparatus, device, and medium provided in this application first construct an initial reward matrix based on the distribution relationship between multiple regional nodes in the target area; second, based on the initial reward matrix, determine the initial cumulative reward value distribution; then, based on the initial cumulative reward value distribution, determine at least two regional nodes among the multiple regional nodes to construct the target planning path for the target moving object; further, after the target moving object moves according to the target planning path, update the initial reward matrix based on the movement state of the target moving object during the movement process to form an updated reward matrix, and determine the updated cumulative reward value distribution based on the updated reward matrix. Based on the above, since the update is based on the movement state of the target moving object during the movement process, prior knowledge can be fully utilized, thereby ensuring a more reliable planning basis in subsequent path planning. Therefore, the reliability of path planning can be improved, thus addressing the problem of relatively low reliability in existing path planning technologies.

[0098] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can also be implemented in other ways. The apparatus and method embodiments described above are merely illustrative. For example, the flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of apparatus, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram and / or flowchart, and combinations of blocks in block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0099] In addition, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0100] If the aforementioned functions are implemented as software functional modules and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, electronic device, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks. It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0101] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A path planning method, characterized in that, include: Based on the distribution relationship between multiple regional nodes in the target area, an initial reward matrix is ​​constructed, wherein each reward parameter in the initial reward matrix is ​​used to reflect the reward obtained from moving from the corresponding first regional node to the second regional node; Based on the initial reward matrix, an initial cumulative reward value distribution is determined, wherein each cumulative reward value in the initial cumulative reward value distribution is used to reflect the cumulative value of the reward obtained from moving from the corresponding first regional node to the second regional node and the reward obtained from moving from the second regional node to other regional nodes; Based on the initial cumulative reward value distribution, at least two regional nodes are determined from the plurality of regional nodes to construct the target planning path of the target moving object; After the target moving object moves based on the target planned path, the initial reward matrix is ​​updated based on the movement state of the target moving object during the movement process to form an updated reward matrix, and the updated cumulative reward value distribution is determined based on the updated reward matrix, wherein the updated cumulative reward value distribution serves as the basis for path planning for the target moving object or other moving objects.

2. The path planning method according to claim 1, characterized in that, The steps of updating the initial reward matrix to form an updated reward matrix based on the motion state of the target moving object during its movement after the target moving object moves according to the target planned path, and determining the updated cumulative reward value distribution based on the updated reward matrix, include: After the target moving object moves based on the target planned path, based on the movement state of the target moving object during the movement process, the congestion characterization parameters between every two adjacent regional nodes in the target planned path are determined; Based on the congestion characterization parameters between every two adjacent regional nodes in the target planning path, the reward parameters between the corresponding regional nodes in the initial reward matrix are updated to form an updated reward matrix, and the updated cumulative reward value distribution is determined based on the updated reward matrix.

3. The path planning method according to claim 2, characterized in that, The step of determining the congestion characterization parameters between every two adjacent regional nodes in the target planning path based on the motion state of the target moving object during its movement after the target moving object has moved according to the target planning path includes: After the target moving object moves based on the target planned path, for every two adjacent regional nodes in the target planned path, the movement speed and path congestion duration of the target moving object between the two adjacent regional nodes are obtained; Based on the movement speed and the path congestion duration, congestion characterization parameters between the two adjacent regional nodes are determined. Specifically, based on the current updated cumulative reward value distribution, at least two regional nodes are identified from among the multiple regional nodes to construct a new target planning path. After the corresponding moving object moves based on the new target planning path, during the update of the current update reward matrix, the determined congestion characterization parameters are the weighted sum of the currently determined congestion characterization parameters and each historically determined congestion characterization parameter. The weighting coefficient decreases as the number of times the moving object moves, representing the reduction in the influence of previous movement conditions on the current situation.

4. The path planning method according to claim 2, characterized in that, The steps of updating the reward parameters between corresponding regional nodes in the initial reward matrix based on the congestion characterization parameters between every two adjacent regional nodes in the target planning path to form an updated reward matrix, and determining the distribution of the updated cumulative reward value based on the updated reward matrix, include: For every two adjacent regional nodes in the target planning path, the difference between the reward parameters between the two adjacent regional nodes and the congestion characterization parameters between the two adjacent regional nodes in the initial reward matrix is ​​calculated to obtain the updated reward parameters, thereby forming the updated reward matrix; The distribution of the updated cumulative reward value is determined based on the updated reward matrix.

5. The path planning method according to any one of claims 1-4, characterized in that, The step of constructing the initial reward matrix based on the distribution relationship between multiple regional nodes in the target region includes: Obtain the distance between any two regional nodes in the target region; Based on the distance between every two region nodes, construct the adjacency matrix of the target region; An initial reward matrix is ​​constructed based on the adjacency matrix. For each pair of connected regional nodes in the target region, a reward parameter between the two regional nodes is determined based on the maximum value in the adjacency matrix and the adjacency parameter between the two regional nodes. For each pair of unconnected regional nodes in the target region, the reward parameter between the two regional nodes is configured as a target value, wherein the target value is less than the reward parameter between any two connected regional nodes.

6. The path planning method according to any one of claims 1-4, characterized in that, The step of determining the initial cumulative reward value distribution based on the initial reward matrix includes: For every two regional nodes in the target region, the reward parameter obtained from moving from the first regional node to the second regional node is obtained from the initial reward matrix to obtain the first reward parameter, and the maximum value of the reward parameter obtained from moving from the second regional node to each other regional node is obtained to obtain the second reward parameter. Based on the first reward parameter, the second reward parameter, and the discount factor configured for the second reward parameter, the cumulative reward value between the two regional nodes is determined; The initial cumulative reward value distribution is obtained based on the cumulative reward value between every two regional nodes in the target region.

7. The path planning method according to any one of claims 1-4, characterized in that, The step of determining at least two region nodes from the plurality of region nodes based on the initial cumulative reward value distribution to construct the target planning path for the target moving object includes: The starting region node and the destination region node of the target moving object are determined from the plurality of region nodes; Using the starting region node as the starting node and the destination region node as the ending node, based on the initial cumulative reward value distribution, the system searches backwards for the node with the maximum reward that has not been visited before, in order to form the target planning path for the target moving object.

8. A path planning device, characterized in that, include: The reward matrix construction module is used to construct an initial reward matrix based on the distribution relationship between multiple regional nodes in the target area. Each reward parameter in the initial reward matrix is ​​used to reflect the reward obtained from moving from the corresponding first regional node to the second regional node. The reward value determination module is used to determine the initial cumulative reward value distribution based on the initial reward matrix, wherein each cumulative reward value in the initial cumulative reward value distribution is used to reflect the cumulative value of the reward obtained from moving from the corresponding first regional node to the second regional node and the reward obtained from moving from the second regional node to other regional nodes; The path planning module is used to determine at least two regional nodes among the multiple regional nodes based on the initial cumulative reward value distribution, so as to construct the target planning path of the target moving object; The reward value update module is used to update the initial reward matrix based on the motion state of the target moving object during the motion process after the target moving object moves based on the target planned path, forming an updated reward matrix, and to determine the updated cumulative reward value distribution based on the updated reward matrix, wherein the updated cumulative reward value distribution serves as the basis for path planning for the target moving object or other moving objects.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor connected to the memory is used to execute a computer program stored in the memory to implement the path planning method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed, performs the path planning method according to any one of claims 1-7.

Images