Unmanned vehicle path planning method, device and equipment and storage medium

Abstract

The application belongs to the technical field of path planning, and discloses an unmanned vehicle path planning method, device, equipment and storage medium. The method comprises the following steps: initializing algorithm parameters and model parameters; establishing a terrain map of an unmanned vehicle travel area, determining obstacle coordinates, unmanned vehicle starting point coordinates and unmanned vehicle ending point coordinates in the terrain map; establishing an initial population based on randomly generating N butterfly individuals; calculating path costs of various population individuals in the initial population and determining a global optimal individual; determining new paths corresponding to various population individuals according to a switching probability in the algorithm parameters; determining a new global optimal individual according to the new paths; judging whether it is necessary to continue to iterate the paths of the population individuals according to a current iteration number and a maximum iteration number, so as to obtain an unmanned vehicle optimal path corresponding to an output global optimal individual. The total path planning cost of the unmanned vehicle can be reduced, and an optimal path can be generated.

Description

Technical Field

[0001] This invention relates to the field of path planning technology, and in particular to a method, apparatus, device, and storage medium for unmanned vehicle path planning. Background Technology

[0002] When using the traditional butterfly optimization algorithm to solve the autonomous vehicle path planning problem, it is easy to suffer from high planning costs, poor accuracy, and getting trapped in local optima. Summary of the Invention

[0003] The main objective of this invention is to provide an unmanned vehicle path planning method, apparatus, device, and storage medium, aiming to solve the technical problems of high planning cost, poor accuracy, and getting trapped in local optima when using the traditional butterfly optimization algorithm to solve the unmanned vehicle path planning problem in the prior art.

[0004] To achieve the above objectives, the present invention provides an unmanned vehicle path planning method, the method comprising the following steps:

[0005] S10. Initialize parameters, wherein the parameters include algorithm parameters and model parameters, the algorithm parameters are the relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are the relevant parameters of the autonomous vehicle path planning model;

[0006] S20. Establish a topographic map of the autonomous vehicle's travel area, and determine the coordinates of obstacles, the starting point coordinates of the autonomous vehicle, and the ending point coordinates of the autonomous vehicle in the topographic map;

[0007] S30. Establish an initial population based on N randomly generated butterfly individuals;

[0008] S40. Based on the model parameters, the obstacle coordinates, the starting coordinates of the unmanned vehicle, and the ending coordinates of the unmanned vehicle, calculate the path cost of each individual in the initial population and determine the globally optimal individual.

[0009] S50. Determine the new path corresponding to each group of individuals based on the switching probability in the algorithm parameters;

[0010] S60. Determine the new globally optimal individual based on the new path;

[0011] S70. Determine if the current iteration count is equal to the maximum iteration count;

[0012] S80. If not, then after using the new global optimal individual as the global optimal individual in step S40, repeat steps S40-S80; if yes, then use the new global optimal individual as the output global optimal individual, and determine the optimal path for the autonomous vehicle based on the output global optimal individual.

[0013] Optionally, determining the new path corresponding to each group individual based on the switching probability in the algorithm parameters includes:

[0014] Generate random values ​​for individuals in various groups;

[0015] Determine whether the random value is less than the switching probability;

[0016] If so, the paths of individuals in the population are updated using the Q-learning global search mechanism to obtain new paths. The Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism.

[0017] If not, the paths of individuals in the population are updated using the local development mechanism of Q-learning to optimize inertia weights, resulting in new paths. The local development mechanism of Q-learning to optimize inertia weights is an improvement on the local development process of the BOA algorithm based on the Q-learning mechanism.

[0018] Optionally, the paths of individuals in various groups are updated using the Q-learning global search mechanism to obtain new paths, including:

[0019] S5031. Define a first state set and a first action set, wherein the first state set includes a first global state, a second global state, and a third global state, and the first action set includes a first global action, a second global action, and a third global action;

[0020] S5032. Based on the first state set and the first action set, generate a first Q table for the population individuals, wherein the first Q table stores the expected reward corresponding to the state-action combination.

[0021] S5033. Determine the first state and first action of an individual in the population, wherein the first action is an action randomly selected from the first action set with probability p and a first maximum action selected with probability 1-p, the first maximum action is the action corresponding to the maximum expected reward in the first Q table, and the probability p is the switching probability.

[0022] S5034. Determine the new expected reward when an individual in the population performs the first selection action in the first state, and update the first Q table according to the new expected reward;

[0023] S5035. Update the path of the population individuals according to the first selection action to obtain the updated path;

[0024] S5036. Determine whether the first iteration number is equal to the first preset iteration number;

[0025] S5037. If not, then after re-determining the first state in step S5033 based on the updated path and re-determining the first selection action in step S5033 based on the updated first Q table, repeat steps S5033-S5037; if yes, then use the updated path as the new path.

[0026] Optionally, determining the first state of an individual in the population includes:

[0027] Determine the previous path cost and current path cost of an individual in the population, wherein the previous path cost is determined based on the path corresponding to the individual in the previous time step, and the current path cost is determined based on the path corresponding to the individual in the current time step.

[0028] The previous path cost is compared with the current path cost to obtain a first comparison result;

[0029] Based on the first comparison result, the first state of the individual in the population is determined from the first state set.

[0030] Optionally, the paths of individuals in various groups are updated using a local development mechanism that optimizes inertia weights through Q-learning, resulting in new paths, including:

[0031] S5041. Define a second set of states and a second set of actions, wherein the second set of states includes a first local state, a second local state, a third local state, and a fourth local state, and the second set of actions includes a first local action, a second local action, a third local action, and a fourth local action.

[0032] S5042. Generate a second Q-table for the population individuals based on the second state set and the second action set;

[0033] S5043. The initial population is divided into subpopulations according to the path cost of individuals in the population, wherein the subpopulations include an elite population, a first ordinary population, a second ordinary population, and an inferior population.

[0034] S5044. Determine the second state of each subpopulation and determine the second choice action of each subpopulation individual, wherein the second choice action is an action randomly selected from the second action set with probability p and a second maximum action selected with probability 1-p, the second maximum action is the action corresponding to the maximum expected reward in the second Q table, and the probability p is the switching probability.

[0035] S5045. Determine the new expected reward when an individual in the population performs the second selection action in the second state, and update the second Q table according to the new expected reward;

[0036] S5046. Update the paths of individuals in each group according to the second selected action to obtain the updated path;

[0037] S5047. Determine whether the second iteration number is equal to the second preset iteration number;

[0038] S5048. If not, then after re-determining the second state in step S5044 based on the updated path and re-determining the second selection action in step S5044 based on the updated second Q table, repeat steps S5044-S5048; if yes, then use the updated path as the new path.

[0039] Optionally, determining the second state of each subpopulation includes:

[0040] The number of individuals in the subpopulation is determined as the first number, and the number of individuals in the subpopulation whose local optimal fitness decreases is determined as the second number.

[0041] The local fitness reduction rate of the subpopulation is determined based on the first quantity and the second quantity;

[0042] The second state of each subpopulation is determined based on the local fitness reduction rate.

[0043] Optionally, determining the new expected reward for an individual in the population to perform the second selection action in the second state, and updating the second Q-table based on the new expected reward, further includes:

[0044] Instant rewards are determined based on the distance between individual populations and the target point;

[0045] When it is determined that the second state is within the first range and its corresponding local fitness reduction rate decreases, the target reward is determined to be a first fixed value;

[0046] When it is determined that the second state is within the second range and its corresponding local fitness reduction rate increases, the target reward is determined to be the second fixed value;

[0047] When it is determined that the local fitness reduction rate corresponding to the second state decreases, the target reward is determined to be a negative immediate reward;

[0048] When it is determined that the local fitness reduction rate corresponding to the second state increases, the target reward is determined to be a positive immediate reward;

[0049] Based on the target reward, determine the new expected reward when an individual in the population performs the second choice action in the second state, and update the second Q table according to the new expected reward.

[0050] Furthermore, to achieve the above objectives, the present invention also proposes an unmanned vehicle path planning device, the unmanned vehicle path planning device comprising:

[0051] An initialization module is used to initialize parameters, wherein the parameters include algorithm parameters and model parameters, the algorithm parameters are relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are relevant parameters of the autonomous vehicle path planning model;

[0052] A module is established to create a terrain map of the autonomous vehicle's travel area and determine the coordinates of obstacles, the starting point coordinates of the autonomous vehicle, and the ending point coordinates of the autonomous vehicle in the terrain map.

[0053] The establishment module is also used to establish an initial population based on N randomly generated butterfly individuals;

[0054] The calculation module is used to calculate the path cost of each individual in the initial population based on the model parameters, the obstacle coordinates, the starting coordinates of the unmanned vehicle, and the ending coordinates of the unmanned vehicle, and to determine the globally optimal individual.

[0055] The determination module is used to determine the new path corresponding to each group of individuals based on the switching probability in the algorithm parameters;

[0056] The determining module is further configured to determine a new globally optimal individual based on the new path;

[0057] The judgment module is used to determine whether the current iteration count is equal to the maximum iteration count;

[0058] The execution module is used to take the new global optimal individual as the output global optimal individual and determine the optimal path for the unmanned vehicle based on the output global optimal individual.

[0059] Furthermore, to achieve the above objectives, the present invention also proposes an unmanned vehicle path planning device, which includes: a memory, a processor, and an unmanned vehicle path planning program stored in the memory and executable on the processor. The unmanned vehicle path planning program is configured to implement the steps of the unmanned vehicle path planning method described above.

[0060] In addition, to achieve the above objectives, the present invention also proposes a storage medium storing an autonomous vehicle path planning program, which, when executed by a processor, implements the steps of the autonomous vehicle path planning method described above.

[0061] The present invention proposes an unmanned vehicle path planning method, apparatus, device, and storage medium. The method involves initializing parameters, including algorithm parameters and model parameters. The algorithm parameters are parameters related to the Q-learning butterfly optimization algorithm, and the model parameters are parameters related to the unmanned vehicle path planning model. The method then establishes a terrain map of the unmanned vehicle's travel area, determining the coordinates of obstacles, the starting point coordinates, and the ending point coordinates of the unmanned vehicle in the terrain map. An initial population is established based on N randomly generated butterfly individuals. Based on the model parameters, the obstacle coordinates, the starting point coordinates, and the ending point coordinates, the path cost of each individual in the initial population is calculated, and the globally optimal individual is determined. A new path is determined for each individual based on the switching probability in the algorithm parameters. A new globally optimal individual is determined based on the new path. The method then determines whether the initial population needs to continue iterating by comparing the current iteration count with the maximum iteration count, thereby determining the output globally optimal individual. Finally, the optimal path for the unmanned vehicle is determined based on the output globally optimal individual. By establishing an autonomous vehicle path planning model, path planning is transformed into a multi-dimensional objective function optimization problem. The Q-learning butterfly optimization algorithm is used to solve the autonomous vehicle path planning problem. The fitness of individual butterflies is evaluated by comprehensively considering the objective function of path length and collision degree. The path planning scheme is iteratively optimized, which can reduce the total cost of autonomous vehicle path planning and generate an optimal path with shorter path and less collision degree. Attached Figure Description

[0062] Figure 1 This is a schematic diagram of the structure of the unmanned vehicle path planning device in the hardware operating environment involved in the embodiments of the present invention;

[0063] Figure 2 This is a flowchart illustrating the first embodiment of the unmanned vehicle path planning method of the present invention;

[0064] Figure 3 This is the terrain map for unmanned vehicle path planning in the first embodiment of the unmanned vehicle path planning method of the present invention;

[0065] Figure 4 This is the butterfly individual encoding diagram in the first embodiment of the unmanned vehicle path planning method of the present invention;

[0066] Figure 5 This is a path planning diagram for an unmanned vehicle in the first embodiment of the unmanned vehicle path planning method of the present invention;

[0067] Figure 6 This is a flowchart illustrating the second embodiment of the unmanned vehicle path planning method of the present invention;

[0068] Figure 7 This is a flowchart illustrating the Q-learning global search mechanism in the second embodiment of the unmanned vehicle path planning method of the present invention.

[0069] Figure 8 This is a schematic diagram of the Q-learning global search mechanism in the second embodiment of the unmanned vehicle path planning method of the present invention;

[0070] Figure 9 This is a flowchart illustrating the local development mechanism for Q-learning to optimize inertia weights in the second embodiment of the unmanned vehicle path planning method of the present invention.

[0071] Figure 10 This is another flowchart illustrating the second embodiment of the unmanned vehicle path planning method of the present invention;

[0072] Figure 11 This is a structural block diagram of the first embodiment of the unmanned vehicle path planning device of the present invention.

[0073] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0074] It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0075] Reference Figure 1 , Figure 1 This is a schematic diagram of the unmanned vehicle path planning device structure in the hardware operating environment involved in the embodiments of the present invention.

[0076] like Figure 1 As shown, the unmanned vehicle path planning device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen and an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wireless-Fidelity (Wi-Fi) interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0077] Those skilled in the art will understand that Figure 1The structure shown does not constitute a limitation on the autonomous vehicle path planning device, and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0078] like Figure 1 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a network communication module, a user interface module, and an unmanned vehicle path planning program.

[0079] exist Figure 1 In the unmanned vehicle path planning device shown, the network interface 1004 is mainly used for data communication with the network server; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the unmanned vehicle path planning device of the present invention can be set in the unmanned vehicle path planning device, and the unmanned vehicle path planning device calls the unmanned vehicle path planning program stored in the memory 1005 through the processor 1001 and executes the unmanned vehicle path planning method provided in the embodiment of the present invention.

[0080] Based on the above hardware structure, an embodiment of the unmanned vehicle path planning method of the present invention is proposed.

[0081] Reference Figure 2 , Figure 2 This is a flowchart illustrating the first embodiment of an unmanned vehicle path planning method according to the present invention.

[0082] In this embodiment, the unmanned vehicle path planning method includes the following steps:

[0083] Step S10: Initialize parameters, wherein the parameters include algorithm parameters and model parameters, the algorithm parameters are the relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are the relevant parameters of the autonomous vehicle path planning model.

[0084] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a mobile phone, tablet computer, or personal computer, or an electronic device or autonomous vehicle path planning device capable of performing the above functions. The following description uses the autonomous vehicle path planning device as an example to illustrate this embodiment and the subsequent embodiments.

[0085] It should be noted that the algorithm parameters include population size, search dimension, maximum number of iterations, switching probability, perception morphology, and incentive index; the model parameters include map environment, starting coordinates and ending coordinates of the autonomous vehicle, number of navigation points, number of obstacles, center coordinates of obstacles, threat radius of obstacles, weight coefficients of the cost function, learning rate, discount rate, and control coefficients.

[0086] Step S20: Establish a topographic map of the autonomous vehicle's travel area, and determine the coordinates of obstacles, the starting point coordinates of the autonomous vehicle, and the ending point coordinates of the autonomous vehicle in the topographic map.

[0087] In the specific implementation, the terrain map is a two-dimensional terrain map, and obstacles can be represented by cylinders on the terrain map, specifically:

[0088] R 2 =(xa) 2 +(xb) 2

[0089] In the formula, (a,b) are the coordinates of the obstacle center, (x,y) are the x and y coordinates of a point on the circumference of the obstacle, and R is the obstacle threat radius.

[0090] In specific implementations, such as Figure 3 As shown, taking a 500m×500m topographic map as an example, the starting coordinates of the unmanned vehicle are set as (0,0) and the ending coordinates of the unmanned vehicle are set as (500,500).

[0091] Step S30: Establish an initial population based on N randomly generated butterfly individuals.

[0092] It should be noted that in the path planning model, let the starting coordinates of the autonomous vehicle be (x0, y0) and the ending coordinates of the autonomous vehicle be (x0, y0). m+1 ,y m+1 Based on the starting and ending coordinates of the autonomous vehicle, the horizontal coordinates of the autonomous vehicle from the starting point to the ending point can be divided into m parts (i.e., setting m navigation points on the terrain map, where m corresponds to the dimension of an individual in the population). The goal of the path planning model is to search for the shortest safe path (i.e., the shortest path from the starting point to the ending point without colliding with any obstacle) using the Q-learning butterfly optimization algorithm. The planned path can be represented as {(x0,y0),(x1,y1),...(x... m ,y m ),(x m+1 ,y m+1 Since the starting and ending coordinates are known, the path to be solved is {(x1,y1),...(x...}. m ,y m )}.

[0093] It should be noted that the coding format of an individual butterfly is as follows: Figure 4 As shown.

[0094] In the specific implementation, Figure 5Taking the topographic map shown as an example, let the number of navigation points m = 4. It can be seen that the horizontal coordinate of the map is divided into 5 equal parts. The x-coordinates of the 4 navigation points are 100 / 200 / 300 / 400 respectively. The key is to determine the y-coordinate value of the navigation points, and y∈[0,500]. Figure 5 These are four autonomous vehicle paths generated under this model, and all four paths are feasible and do not collide with any obstacles.

[0095] Step S40: Based on the model parameters, obstacle coordinates, driverless vehicle starting point coordinates, and driverless vehicle ending point coordinates, calculate the path cost of each individual in the initial population and determine the globally optimal individual.

[0096] It is understandable that the individual with the lowest path cost in the initial population is the globally optimal individual.

[0097] It should be noted that the path cost of an individual in the population is determined by the path length and the collision severity (collision severity refers to the degree of collision between the autonomous vehicle and obstacles). The formula for calculating the path cost is as follows:

[0098] Cost=w1×Length+w2×Collision

[0099] In the formula, w1 and w2 are weighting coefficients (weighting coefficients can be used to define the emphasis on path length and collision degree during path planning), and 0 <w1、w2<1。

[0100] In the specific implementation, the formula for calculating the path length is as follows:

[0101]

[0102] In the specific implementation, the formula for calculating the collision level is as follows:

[0103]

[0104]

[0105] In the formula, K represents the number of obstacles, h(k) is the collision degree between the UAV and the kth obstacle, (a,b) are the center coordinates of the obstacle, and R is the obstacle threat radius.

[0106] It should be noted that in the traditional Butterfly Optimization Algorithm (BOA algorithm), each butterfly emits a certain concentration of fragrance. After diffusing through the air, other butterflies are attracted to it based on the fragrance concentration. If a butterfly smells the highest concentration of fragrance, it will move towards that fragrance; this approaching process is the global search. If a butterfly does not detect any fragrance, it will fly randomly in space; this random walk process is the local exploration. The formula for calculating the fragrance concentration emitted by a butterfly is:

[0107] f = c·I a

[0108] In the formula, I is the stimulus intensity, c is the perceptual morphology, a is the stimulation index (the stimulation index can be understood as the absorption of different degrees of fragrance), and a and c ∈ [0,1].

[0109] It should be noted that in the BOA algorithm, a is usually set to 0.1, and the initial value of c is 0.01. The iterative change formula is as follows:

[0110]

[0111] In the formula, Tmax is the maximum number of iterations.

[0112] It should be noted that in traditional methods, the position update method for individual butterflies during global search is as follows:

[0113] x i (t+1)=x i (t)+(r1 2 ×g * -x i (t))×f i

[0114] In the formula, x i (t) represents the position of the i-th butterfly in the population at the t-th iteration, x i (t+1) represents the position of the i-th butterfly in the population at the (t+1)-th iteration, g * Let r1 ∈ [0,1] be the current optimal position, and f be a random variable. i Let be the fragrance concentration of the i-th butterfly in the population.

[0115] It should be noted that in traditional methods, the position update method for individual butterflies during local development is as follows:

[0116] x i (t+1)=x i (t)+(r2 2 ×x j (t)-x k (t))×f i

[0117] In the formula, x j (t), x k (t) represents the positions of two randomly selected butterflies in the population at the t-th iteration, where r2∈[0,1] is a random variable.

[0118] Step S50: Determine the new path corresponding to each group of individuals based on the switching probability in the algorithm parameters.

[0119] In the specific implementation, a random value needs to be generated, and then compared with the switching probability to determine how to update the path corresponding to the individual in the population. Specifically, when the random value is less than or equal to the switching probability, the path of the individual in the population is updated using the Q-learning global search mechanism to obtain a new path. The Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism. When the random value is greater than the switching probability, the path of the individual in the population is updated using the Q-learning local development mechanism for optimizing inertia weights to obtain a new path. The Q-learning local development mechanism for optimizing inertia weights is an improvement on the local development process of the BOA algorithm based on the Q-learning mechanism.

[0120] Step S60: Determine the new globally optimal individual based on the new path.

[0121] It should be noted that after all individuals in the initial population have updated their paths, the cost of the new path for each individual can be determined based on their new path. The individual with the lowest cost of the new path is the new globally optimal individual.

[0122] Step S70: Determine whether the current iteration number is equal to the maximum iteration number.

[0123] It should be noted that when the paths of all individuals in the initial population are updated simultaneously, it can be recorded as one iteration of the initial population. Each time the initial population iterates, the current iteration count is incremented by 1.

[0124] Step S80: If not, then after using the new global optimal individual as the global optimal individual in step S40, repeat steps S40-S80; if yes, then use the new global optimal individual as the output global optimal individual, and determine the optimal path for the autonomous vehicle based on the output global optimal individual.

[0125] In practice, the optimal path for the autonomous vehicle can be determined by decoding the output of the globally optimal individual.

[0126] This embodiment initializes parameters, including algorithm parameters and model parameters. The algorithm parameters are relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are relevant parameters of the autonomous vehicle path planning model. A terrain map of the autonomous vehicle's travel area is established, and the coordinates of obstacles, the starting point coordinates, and the ending point coordinates of the autonomous vehicle are determined in the terrain map. An initial population is established based on N randomly generated butterfly individuals. Based on the model parameters, the obstacle coordinates, the starting point coordinates, and the ending point coordinates, the path cost of each individual in the initial population is calculated, and the globally optimal individual is determined. New paths corresponding to each individual are determined according to the switching probabilities in the algorithm parameters. New globally optimal individuals are determined based on the new paths. Whether the initial population needs to continue iterating is determined by comparing the current iteration count with the maximum iteration count, thereby determining the output globally optimal individual. Finally, the optimal path for the autonomous vehicle is determined based on the output globally optimal individual. By establishing an autonomous vehicle path planning model, path planning is transformed into a multi-dimensional objective function optimization problem. The Q-learning butterfly optimization algorithm is used to solve the autonomous vehicle path planning problem. The fitness of individual butterflies is evaluated by comprehensively considering the objective function of path length and collision degree. The path planning scheme is iteratively optimized, which can reduce the total cost of autonomous vehicle path planning and generate an optimal path with shorter path and less collision degree.

[0127] refer to Figure 6 , Figure 6 This is a flowchart illustrating a second embodiment of an unmanned vehicle path planning method according to the present invention.

[0128] Based on the first embodiment described above, the autonomous vehicle path planning method in this embodiment determines the new path corresponding to each group of individuals according to the switching probability in the algorithm parameters, including:

[0129] Step S501: Generate random values ​​for individuals in various groups.

[0130] It is understandable that random values ​​refer to randomly generated numerical values, and the random values ​​of individuals in different populations can be the same or different.

[0131] Step S502: Determine whether the random value is less than the switching probability.

[0132] It should be noted that the path update method for individuals in the population can be determined by comparing random values ​​with switching probabilities.

[0133] Step S503: If yes, then update the paths of individuals in the population using the Q-learning global search mechanism to obtain new paths, wherein the Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism.

[0134] In one embodiment, such as Figure 7 As shown, the Q-learning global search mechanism is used to update the paths of individuals in various groups, resulting in new paths, including:

[0135] S5031. Define a first state set and a first action set, wherein the first state set includes a first global state, a second global state, and a third global state, and the first action set includes a first global action, a second global action, and a third global action.

[0136] S5032. Based on the first state set and the first action set, generate a first Q table for the population individuals, wherein the first Q table stores the expected reward corresponding to the state-action combination.

[0137] In the specific implementation, the first Q table is shown in the following table:

[0138]

[0139] In the table, 'a' represents the global action, 'S' represents the global state, and 'Q' represents the expected reward.

[0140] S5033. Determine the first state and first action of an individual in the population, wherein the first action is an action randomly selected from the first action set with probability p and a first maximum action selected with probability 1-p, the first maximum action being the action corresponding to the maximum expected reward in the first Q table, and the probability p being the switching probability.

[0141] It should be noted that the first state is any state in the first set of states.

[0142] In one embodiment, determining the first state of an individual in the population includes:

[0143] Determine the previous path cost and current path cost of an individual in the population, wherein the previous path cost is determined based on the path corresponding to the individual in the previous time step, and the current path cost is determined based on the path corresponding to the individual in the current time step.

[0144] The previous path cost is compared with the current path cost to obtain a first comparison result;

[0145] Based on the first comparison result, the first state of the individual in the population is determined from the first state set.

[0146] It is understandable that the path cost is inversely proportional to the quality of the path for an individual in the population; the smaller the path cost, the better the path for the individual in the population, and the larger the path cost, the worse the path for the individual in the population.

[0147] It should be noted that the first state of a population individual can be determined from the first state set based on the path cost of the individual. Specifically, the first global state indicates that the generated individual is better than the original individual (i.e., the path cost of the path corresponding to the population individual at a previous time is greater than the path cost of the path corresponding to the population individual at the current time), the second global state indicates that the generated individual is the same as the original individual (i.e., the path cost of the path corresponding to the population individual at a previous time is equal to the path cost of the path corresponding to the population individual at the current time), and the third global state indicates that the generated individual is worse than the original individual (i.e., the path cost of the path corresponding to the population individual at a previous time is less than the path cost of the path corresponding to the population individual at the current time).

[0148] Understandably, when the first comparison result is that the cost of the previous path is greater than the cost of the current path, it means that the individual in the population is in the first global state; when the first comparison result is that the cost of the previous path is equal to the cost of the current path, it means that the individual in the population is in the second global state; and when the first comparison result is that the cost of the previous path is less than the cost of the current path, it means that the individual in the population is in the third global state.

[0149] In the specific implementation, at the current moment, when an individual in the population is in the first global state, it receives a reward of 1; at the current moment, when an individual in the population is in the second global state, it receives a reward of -0.001; and at the current moment, when an individual in the population is in the third global state, it receives a reward of -1, as specifically represented below:

[0150] In the formula, R is the reward value.

[0151] It should be noted that the first selection action is determined based on a p-greedy strategy. The first selection action is a random action selected from the first action set with probability p and a first maximum action selected with probability 1-p. The first maximum action is the action in the first Q table that has the maximum expected reward.

[0152] In a practical implementation, the probability corresponding to the first chosen action can be defined as:

[0153]

[0154] In the formula, π(a|s) is the probability of choosing action a in state s, m is the number of actions in action set A in state s, where A = 3, and a* is the first maximum action.

[0155] S5034. Determine the new expected reward when an individual in the population performs the first selection action in the first state, and update the first Q table according to the new expected reward.

[0156] It should be noted that determining the new expected reward for an individual in the population to perform the first selection action in the first state, and updating the first Q table based on the new expected reward, specifically involves:

[0157]

[0158] In the formula, s t Let s be the state of an individual in the population at the current time t. t+1 Let a be the state of an individual in the population at the next time step t+1. t Let r be the action performed by an individual in the population at the current time t, β be the system learning rate, γ be the discount rate, and 0 ≤ β, γ ≤ 1. t+1 MaxQ(s) represents the immediate reward value obtained by the agent after performing an action in state st (i.e., the reward value of an individual in the population under different states). t+1 ,a t ) is in state s t+1 Next, execute action a t The maximum expected reward obtained, Q t+1 (s t+1 ,a t γ represents the new expected reward of an individual in the population when performing the first choice action in the first state. If γ is large and close to 1, it indicates that the individual in the population focuses on future rewards; if γ is small and close to 0, it indicates that the individual in the population focuses on the immediate reward obtained at the current moment.

[0159] S5035. Update the path of the population individuals according to the first selection action to obtain the updated path.

[0160] It should be noted that, as Figure 8 The first global action shown is the Differential Evolutionary Mutation (DE) strategy, the second global action is the global search strategy of the Flower Pollination Algorithm (FPA), and the third global action is the global search strategy of the original Butterfly Optimization Algorithm (BOA).

[0161] In the specific implementation, the differential evolution mutation (DE) strategy is used to update the path formula for individuals in the population, as follows:

[0162] x i (t+1)=x rand1 (t)+F·(x rand2 (t)-x rand3 (t))

[0163] In the formula, x rand1 x rand2 x rand3 These refer to three individuals randomly selected from the initial population during the t-th iteration, where F is the mutation operator, typically 0 ≤ F ≤ 1, and x i(t+1) represents the path of the i-th individual in the initial population during the (t+1)-th iteration.

[0164] In the specific implementation, the global search strategy of the Flower Pollination Algorithm (FPA) is used to update the paths of individuals in the population as follows:

[0165] x i (t+1)=x i (t)+β·L·(x i (t)-x best (t))

[0166] In the formula, x i (t+1) represents the path of the i-th individual in the initial population during the (t+1)-th iteration, x i (t) represents the path of the i-th individual in the initial population at the t-th iteration, x best (t) represents the globally optimal individual in the initial population at the t-th iteration, β is the step size control factor, and L is the Levy flight step size.

[0167] In the specific implementation, the global search strategy of the original Butterfly Optimization Algorithm (BOA) is used to update the paths of individuals in the population as follows:

[0168] x i (t+1)=x i (t)+(r1 2 ×g * -x i (t))×f i

[0169] In the formula, x i (t+1) represents the path of the i-th individual in the initial population during the (t+1)-th iteration, x i (t) represents the path of the i-th individual in the initial population at the t-th iteration, g * Let r1 be the globally optimal individual in the initial population at the t-th iteration, and let f be a random variable between [0,1]. i The fragrance concentration is the i-th individual in the initial population.

[0170] It should be noted that in the traditional Butterfly Optimization Algorithm (BOA algorithm), each butterfly emits a certain concentration of fragrance. After diffusing through the air, other individuals are attracted to it based on the fragrance concentration. If a butterfly smells the highest concentration of fragrance, it will move towards that fragrance; this approaching process is the global search. If a butterfly (i.e., an individual in the population) does not perceive any fragrance, it will randomly wander through the space; this random wandering process is the local exploration. The formula for calculating the fragrance concentration emitted by a butterfly is:

[0171] f = c·Ia

[0172] In the formula, I is the stimulus intensity, c is the perceptual morphology, a is the stimulation index (the stimulation index can be understood as the absorption of different degrees of fragrance), and a and c ∈ [0,1].

[0173] It should be noted that in the BOA algorithm, a is usually set to 0.1, and the initial value of c is 0.01. The iterative change formula is as follows:

[0174]

[0175] In the formula, Tmax is the maximum number of iterations.

[0176] S5036. Determine whether the first iteration number is equal to the first preset iteration number.

[0177] S5037. If not, then after re-determining the first state in step S5033 based on the updated path and re-determining the first selection action in step S5033 based on the updated first Q table, repeat steps S5033-S5037; if yes, then use the updated path as the new path.

[0178] It's important to note that when using the Q-learning butterfly optimization algorithm to update the paths of individuals in the population, each butterfly individual (population individual) corresponds to an agent in the Q-learning system. The individual's search strategy (path update method) corresponds to the action performed by the agent, and the algorithm's iteration result corresponds to the Q-learning state. The Q-learning butterfly optimization algorithm establishes an independent Q-table for each individual in the population. Individuals select actions based on their environment and state, thereby improving the algorithm's global search capability. For a given action 'a' chosen by an individual, the individual receives feedback R and the next state 's', and updates its Q-table. If the individual's fitness is optimized at this point, the environmental feedback from the state-action pair is positive, and the corresponding Q-value increases; if the individual's fitness is not optimized, the environmental feedback from the state-action pair is negative, and the corresponding Q-value decreases.

[0179] Step S504: If not, update the path of the individual in the population using the local development mechanism of Q-learning to optimize the inertia weight, and obtain a new path. The local development mechanism of Q-learning to optimize the inertia weight is obtained by improving the local development process of the BOA algorithm based on the Q-learning mechanism.

[0180] In one embodiment, such as Figure 9 As shown, the path of individuals in various groups is updated using the local exploitation mechanism of Q-learning to optimize inertia weights, resulting in new paths, including:

[0181] S5041. Define a second set of states and a second set of actions, wherein the second set of states includes a first local state, a second local state, a third local state, and a fourth local state, and the second set of actions includes a first local action, a second local action, a third local action, and a fourth local action.

[0182] S5042. Generate a second Q table for the population individuals based on the second state set and the second action set.

[0183] In the specific implementation, the second Q table is shown in the following table:

[0184]

[0185] In the table, 'a' represents a local action, 'S' represents a local state, and 'Q' represents the expected reward.

[0186] S5043. The initial population is divided into subpopulations based on the path cost of individuals in the population, wherein the subpopulations include an elite population, a first ordinary population, a second ordinary population, and an inferior population.

[0187] In the specific implementation, the initial population is divided into an elite population, a first ordinary population, a second ordinary population, and an inferior population in a 1:4:4:1 ratio based on the path cost of the individuals. If the number of individuals in the initial population is N, then the number of individuals in the elite population is N / 10, the number of individuals in both ordinary populations is 4N / 10, and the number of individuals in the inferior population is N / 10. Generally speaking, the globally optimal individual is more likely to be found in the elite population. Enhancing the local exploitation ability of the elite population can increase the probability of finding the globally optimal solution (i.e., the globally optimal individual). The ordinary population only needs to maintain a balanced global search and local exploitation ability. The inferior population is generally far from the globally optimal solution, so its global search ability should be enhanced to allow the algorithm to obtain more high-quality solutions.

[0188] S5044. Determine the second state of each subpopulation and determine the second choice action of each subpopulation individual, wherein the second choice action is an action randomly selected from the second action set with probability p and a second maximum action selected with probability 1-p, the second maximum action is the action corresponding to the maximum expected reward in the second Q table, and the probability p is the switching probability.

[0189] It should be noted that the second state is any state in the set of the second states.

[0190] It should be noted that the second selection action is determined based on a p-greedy strategy. The second selection action is an action randomly selected from the second action set with probability p and a second maximum action selected with probability 1-p. The maximum action is the action in the second Q table that has the maximum expected reward.

[0191] In a practical implementation, the probability corresponding to the second choice action can be defined as:

[0192]

[0193] In the formula, π(a|s) is the probability of choosing action a in state s, m is the number of actions in action set A in state s, where A = 4, and a* is the second largest action.

[0194] It should be noted that the actions performed by individuals in the population correspond to increasing or decreasing the inertia weight. The step size for increasing or decreasing the weight is set to 0.05 or 0.1 depending on the state, and four actions are set for each individual, with the action set A = {a1, a2, a3, a4}, as shown in Table 3. Generally speaking, when the proportion of individuals in the population whose local fitness reduction rate decreases is high, the individuals should decrease the inertia weight value to accelerate the convergence of the algorithm; while when the proportion of individuals in the population whose local fitness reduction rate decreases is low, the individuals should increase the inertia weight value to expand the population search range and escape the local optimum as early as possible.

[0195] In the actual implementation, the action value corresponding to the action is set according to different states:

[0196]

[0197] In one embodiment, determining the second state of each subpopulation includes:

[0198] The number of individuals in the subpopulation is determined as the first number, and the number of individuals in the subpopulation whose local optimal fitness decreases is determined as the second number.

[0199] The local fitness reduction rate of the subpopulation is determined based on the first quantity and the second quantity;

[0200] The second state of each subpopulation is determined based on the local fitness reduction rate.

[0201] In the specific implementation, the correspondence between the range of PF (local fitness reduction rate) and the first local state, second local state, third local state, and fourth local state is shown in the table below, where s1 represents the first local state, s2 represents the second local state, s3 represents the third local state, and s4 represents the fourth local state:

[0202]

[0203] It should be noted that the formula for calculating the population's PF (local fitness reduction rate) is as follows:

[0204] PF = N' / N

[0205] In the formula, N is the first quantity, N' is the second quantity, and PF represents the local fitness reduction rate.

[0206] Understandably, after determining the local fitness reduction rate of the subpopulation, the state of the subpopulation (i.e., the second state) can be determined based on the range corresponding to that local fitness reduction rate.

[0207] S5045. Determine the new expected reward when an individual in the population performs the second selection action in the second state, and update the second Q table according to the new expected reward.

[0208] In one embodiment, determining the new expected reward when an individual in the population performs the second selection action in the second state, and updating the second Q-table based on the new expected reward, further includes:

[0209] Instant rewards are determined based on the distance between individual populations and the target point;

[0210] When it is determined that the second state is within the first range and its corresponding local fitness reduction rate decreases, the target reward is determined to be a first fixed value;

[0211] When it is determined that the second state is within the second range and its corresponding local fitness reduction rate increases, the target reward is determined to be the second fixed value;

[0212] When it is determined that the local fitness reduction rate corresponding to the second state decreases, the target reward is determined to be a negative immediate reward;

[0213] When it is determined that the local fitness reduction rate corresponding to the second state increases, the target reward is determined to be a positive immediate reward;

[0214] Based on the target reward, determine the new expected reward when an individual in the population performs the second choice action in the second state, and update the second Q table according to the new expected reward.

[0215] It should be noted that the formula for calculating the immediate reward based on the distance between an individual in the population and the target point is as follows:

[0216]

[0217] In the formula, dis is the distance between an individual in the population and the target point, calculated using Euclidean distance; λ and ζ are control coefficients; a represents the second choice action; and s represents the second state.

[0218] In practical implementation, the formula for defining the target reward is as follows:

[0219]

[0220] In the formula, R represents the target reward, PF represents the local fitness reduction rate, the first fixed value is -10, and the second fixed value is 10.

[0221] It should be noted that, based on the target reward, a new expected reward is determined when an individual in the population performs the second choice action in the second state, and the second Q table is updated according to the new expected reward, specifically as follows:

[0222]

[0223] In the formula, s t Let s be the state of an individual in the population at the current time t. t+1 Let a be the state of an individual in the population at the next time step t+1. t Let r be the action performed by an individual in the population at the current time t, β be the system learning rate, γ be the discount rate, and 0 ≤ β, γ ≤ 1. t+1 For an individual in a population in state s t The immediate reward value obtained after performing the action (i.e., the target reward corresponding to an individual in the population under different states), maxQ(s) t+1 ,a t ) is in state s t+1 Next, execute action a t The maximum expected reward, Q t+1 (s t+1 ,a t ) is the new expected reward for an individual in the population when performing the second choice action in the second state.

[0224] S5046. Update the paths of various group individuals according to the second selection action to obtain the updated path.

[0225] S5047. Determine whether the second iteration number is equal to the second preset iteration number.

[0226] S5048. If not, then after re-determining the second state in step S5044 based on the updated path and re-determining the second selection action in step S5044 based on the updated second Q table, repeat steps S5044-S5048; if yes, then use the updated path as the new path.

[0227] In practical implementation, the Q-learning butterfly optimization algorithm is used to solve the autonomous vehicle path planning problem, such as... Figure 10 As shown.

[0228] This embodiment generates random values ​​for individuals in various population groups; it then determines whether the random value is less than the switching probability; if so, it updates the paths of the individuals in the population using a Q-learning global search mechanism to obtain a new path. This Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism. If not, it updates the paths of the individuals in the population using a Q-learning-optimized inertia weight local development mechanism to obtain a new path. This Q-learning-optimized inertia weight local development mechanism is an improvement on the local development process of the BOA algorithm based on the Q-learning mechanism. Through this approach, a Q-learning global search mechanism is designed, enabling the generated path planning scheme to make the action selection with the optimal reward based on the current environment and state. Simultaneously, it enriches the diversity of the population by incorporating different individual update methods. Furthermore, a Q-learning-optimized inertia weight local development mechanism is designed, using the Q-learning algorithm to adaptively adjust the inertia weight of the updated individuals in the population, effectively balancing the relationship between search and development, thereby improving the overall search accuracy of the algorithm.

[0229] Furthermore, this embodiment of the invention also proposes a storage medium storing an autonomous vehicle path planning program, which, when executed by a processor, implements the steps of the autonomous vehicle path planning method described above.

[0230] Reference Figure 11 , Figure 11 This is a structural block diagram of the first embodiment of the unmanned vehicle path planning device of the present invention.

[0231] like Figure 11 As shown, the unmanned vehicle path planning device proposed in this embodiment of the invention includes:

[0232] Initialization module 10 is used to initialize parameters, wherein the parameters include algorithm parameters and model parameters, the algorithm parameters are relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are relevant parameters of the autonomous vehicle path planning model;

[0233] Module 20 is used to create a terrain map of the autonomous vehicle's travel area and determine the coordinates of obstacles, the starting point coordinates of the autonomous vehicle, and the ending point coordinates of the autonomous vehicle in the terrain map.

[0234] The establishment module 20 is also used to establish an initial population based on N randomly generated butterfly individuals;

[0235] The calculation module 30 is used to calculate the path cost of each individual in the initial population based on the model parameters, the obstacle coordinates, the starting coordinates of the unmanned vehicle, and the ending coordinates of the unmanned vehicle, and to determine the globally optimal individual.

[0236] The determination module 40 is used to determine the new path corresponding to each group of individuals based on the switching probability in the algorithm parameters;

[0237] The determining module 40 is further configured to determine a new globally optimal individual based on the new path;

[0238] Module 50 is used to determine whether the current iteration count is equal to the maximum iteration count.

[0239] The execution module 60 is used to take the new global optimal individual as the output global optimal individual and determine the optimal path of the unmanned vehicle based on the output global optimal individual.

[0240] It should be understood that the above are merely illustrative examples and do not constitute any limitation on the technical solutions of the present invention. In specific applications, those skilled in the art can make settings as needed, and the present invention does not impose any restrictions on this.

[0241] This embodiment initializes parameters, including algorithm parameters and model parameters. The algorithm parameters are relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are relevant parameters of the autonomous vehicle path planning model. A terrain map of the autonomous vehicle's travel area is established, and the coordinates of obstacles, the starting point coordinates, and the ending point coordinates of the autonomous vehicle are determined in the terrain map. An initial population is established based on N randomly generated butterfly individuals. Based on the model parameters, the obstacle coordinates, the starting point coordinates, and the ending point coordinates, the path cost of each individual in the initial population is calculated, and the globally optimal individual is determined. New paths corresponding to each individual are determined according to the switching probabilities in the algorithm parameters. New globally optimal individuals are determined based on the new paths. Whether the initial population needs to continue iterating is determined by comparing the current iteration count with the maximum iteration count, thereby determining the output globally optimal individual. Finally, the optimal path for the autonomous vehicle is determined based on the output globally optimal individual. By establishing an autonomous vehicle path planning model, path planning is transformed into a multi-dimensional objective function optimization problem. The Q-learning butterfly optimization algorithm is used to solve the autonomous vehicle path planning problem. The fitness of individual butterflies is evaluated by comprehensively considering the objective function of path length and collision degree. The path planning scheme is iteratively optimized, which can reduce the total cost of autonomous vehicle path planning and generate an optimal path with shorter path and less collision degree.

[0242] In one embodiment, the determining module 40 is further configured to:

[0243] Generate random values ​​for individuals in various groups;

[0244] Determine whether the random value is less than the switching probability;

[0245] If so, the paths of individuals in the population are updated using the Q-learning global search mechanism to obtain new paths. The Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism.

[0246] If not, the paths of individuals in the population are updated using the local development mechanism of Q-learning to optimize inertia weights, resulting in new paths. The local development mechanism of Q-learning to optimize inertia weights is an improvement on the local development process of the BOA algorithm based on the Q-learning mechanism.

[0247] In one embodiment, the determining module 40 is further configured to:

[0248] Define a first set of states and a first set of actions, wherein the first set of states includes a first global state, a second global state, and a third global state, and the first set of actions includes a first global action, a second global action, and a third global action;

[0249] Based on the first set of states and the first set of actions, a first Q table is generated for each individual in the population, wherein the first Q table stores the expected reward corresponding to the state-action combination.

[0250] Determine the first state and first action of an individual in the population, wherein the first action is an action randomly selected from the first action set with probability p and a first maximum action selected with probability 1-p, the first maximum action being the action corresponding to the maximum expected reward in the first Q table, and the probability p being the switching probability.

[0251] Determine the new expected reward for an individual in the population when performing the first selection action in the first state, and update the first Q table based on the new expected reward;

[0252] The updated path is obtained by updating the path of the individuals in the population based on the first selected action;

[0253] Determine whether the first iteration count is equal to the first preset iteration count;

[0254] Based on the judgment results, determine whether it is necessary to repeat the iteration to update the paths of individuals in the population.

[0255] In one embodiment, the determining module 40 is further configured to:

[0256] Determine the previous path cost and current path cost of an individual in the population, wherein the previous path cost is determined based on the path corresponding to the individual in the previous time step, and the current path cost is determined based on the path corresponding to the individual in the current time step.

[0257] The previous path cost is compared with the current path cost to obtain a first comparison result;

[0258] Based on the first comparison result, the first state of the individual in the population is determined from the first state set.

[0259] In one embodiment, the determining module 40 is further configured to:

[0260] Define a second set of states and a second set of actions, wherein the second set of states includes a first local state, a second local state, a third local state, and a fourth local state, and the second set of actions includes a first local action, a second local action, a third local action, and a fourth local action;

[0261] Based on the second set of states and the second set of actions, a second Q-table is generated for each individual in the population.

[0262] The initial population is divided into subpopulations based on the path cost of individuals in the population, wherein the subpopulations include an elite population, a first ordinary population, a second ordinary population, and an inferior population.

[0263] Determine the second state of each subpopulation and determine the second choice action of each subpopulation individual, wherein the second choice action is an action randomly selected from the second action set with probability p and a second maximum action selected with probability 1-p, the second maximum action is the action corresponding to the maximum expected reward in the second Q table, and the probability p is the switching probability;

[0264] Determine the new expected reward for an individual in the population to perform the second choice action in the second state, and update the second Q table based on the new expected reward;

[0265] The updated path is obtained by updating the paths of individuals in various groups based on the second selected action;

[0266] Determine whether the second iteration number is equal to the second preset iteration number;

[0267] Based on the judgment results, determine whether it is necessary to repeat the iteration to update the paths of individuals in the population.

[0268] In one embodiment, the determining module 40 is further configured to:

[0269] The number of individuals in the subpopulation is determined as the first number, and the number of individuals in the subpopulation whose local optimal fitness decreases is determined as the second number.

[0270] The local fitness reduction rate of the subpopulation is determined based on the first quantity and the second quantity;

[0271] The second state of each subpopulation is determined based on the local fitness reduction rate.

[0272] In one embodiment, the determining module 40 is further configured to:

[0273] Instant rewards are determined based on the distance between individual populations and the target point;

[0274] When it is determined that the second state is within the first range and its corresponding local fitness reduction rate decreases, the target reward is determined to be a first fixed value;

[0275] When it is determined that the second state is within the second range and its corresponding local fitness reduction rate increases, the target reward is determined to be the second fixed value;

[0276] When it is determined that the local fitness reduction rate corresponding to the second state decreases, the target reward is determined to be a negative immediate reward;

[0277] When it is determined that the local fitness reduction rate corresponding to the second state increases, the target reward is determined to be a positive immediate reward;

[0278] Based on the target reward, determine the new expected reward when an individual in the population performs the second choice action in the second state, and update the second Q table according to the new expected reward.

[0279] It should be noted that the workflow described above is merely illustrative and does not limit the scope of protection of this invention. In practical applications, those skilled in the art can select some or all of the workflow to achieve the purpose of this embodiment according to actual needs, and no restrictions are imposed here.

[0280] In addition, for technical details not described in detail in this embodiment, please refer to the unmanned vehicle path planning method provided in any embodiment of the present invention, which will not be repeated here.

[0281] Furthermore, it should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0282] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0283] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as read-only memory (ROM) / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0284] The above are merely preferred embodiments of the present invention and do not limit the scope of the patent. Any equivalent structural or procedural transformations made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the scope of patent protection of the present invention.

Claims

1. A method for unmanned vehicle path planning, characterized in that, The autonomous vehicle path planning method includes: S10. Initialize parameters, wherein the parameters include algorithm parameters and model parameters, the algorithm parameters are the relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are the relevant parameters of the autonomous vehicle path planning model; S20. Establish a topographic map of the autonomous vehicle's travel area, and determine the coordinates of obstacles, the starting point coordinates of the autonomous vehicle, and the ending point coordinates of the autonomous vehicle in the topographic map; S30. Establish an initial population based on N randomly generated butterfly individuals; S40. Based on the model parameters, the obstacle coordinates, the starting coordinates of the unmanned vehicle, and the ending coordinates of the unmanned vehicle, calculate the path cost of each individual in the initial population and determine the globally optimal individual. S50. Determine the new path corresponding to each group of individuals based on the switching probability in the algorithm parameters; S60. Determine the new globally optimal individual based on the new path; S70. Determine if the current iteration count is equal to the maximum iteration count; S80. If not, then after using the new global optimal individual as the global optimal individual in step S40, repeat steps S40-S80; if yes, then use the new global optimal individual as the output global optimal individual, and determine the optimal path for the unmanned vehicle based on the output global optimal individual. The step of determining the new path corresponding to each group individual based on the switching probability in the algorithm parameters includes: Generate random values ​​for individuals in various groups; Determine whether the random value is less than the switching probability; If so, the paths of individuals in the population are updated using the Q-learning global search mechanism to obtain new paths. The Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism. If not, the path of the individual in the population is updated by using the local development mechanism of Q-learning to optimize the inertia weight, and a new path is obtained. The local development mechanism of Q-learning to optimize the inertia weight is obtained by improving the local development process of the BOA algorithm based on the Q-learning mechanism. The step of using the Q-learning global search mechanism to update the paths of individuals in various groups to obtain new paths includes: S5031. Define a first state set and a first action set, wherein the first state set includes a first global state, a second global state, and a third global state, and the first action set includes a first global action, a second global action, and a third global action; S5032. Based on the first state set and the first action set, generate a first Q table for the population individuals, wherein the first Q table stores the expected reward corresponding to the state-action combination. S5033. Determine the first state and first action of an individual in the population, wherein the first action is an action randomly selected from the first action set with probability p and a first maximum action selected with probability 1-p, the first maximum action is the action corresponding to the maximum expected reward in the first Q table, and the probability p is the switching probability. S5034. Determine the new expected reward when an individual in the population performs the first selection action in the first state, and update the first Q table according to the new expected reward; S5035. Update the path of the population individuals according to the first selection action to obtain the updated path; S5036. Determine whether the first iteration number is equal to the first preset iteration number; S5037. If not, then after re-determining the first state in step S5033 based on the updated path and re-determining the first selection action in step S5033 based on the updated first Q table, repeat steps S5033-S5037; if yes, then use the updated path as the new path.

2. The method as described in claim 1, characterized in that, Determining the first state of an individual in the population includes: Determine the previous path cost and current path cost of an individual in the population, wherein the previous path cost is determined based on the path corresponding to the individual in the previous time step, and the current path cost is determined based on the path corresponding to the individual in the current time step. The previous path cost is compared with the current path cost to obtain a first comparison result; Based on the first comparison result, the first state of the individual in the population is determined from the first state set.

3. The method as described in claim 1, characterized in that, By utilizing Q-learning to optimize the local exploitation mechanism of inertia weights, the paths of individuals in various groups are updated to obtain new paths, including: S5041. Define a second set of states and a second set of actions, wherein the second set of states includes a first local state, a second local state, a third local state, and a fourth local state, and the second set of actions includes a first local action, a second local action, a third local action, and a fourth local action. S5042. Generate a second Q-table for the population individuals based on the second state set and the second action set; S5043. The initial population is divided into subpopulations according to the path cost of individuals in the population, wherein the subpopulations include an elite population, a first ordinary population, a second ordinary population, and an inferior population. S5044. Determine the second state of each subpopulation and determine the second choice action of each subpopulation individual, wherein the second choice action is an action randomly selected from the second action set with probability p and a second maximum action selected with probability 1-p, the second maximum action is the action corresponding to the maximum expected reward in the second Q table, and the probability p is the switching probability. S5045. Determine the new expected reward when an individual in the population performs the second selection action in the second state, and update the second Q table according to the new expected reward; S5046. Update the paths of individuals in each group according to the second selected action to obtain the updated path; S5047. Determine whether the second iteration number is equal to the second preset iteration number; S5048. If not, then after re-determining the second state in step S5044 based on the updated path and re-determining the second selection action in step S5044 based on the updated second Q table, repeat steps S5044-S5048; if yes, then use the updated path as the new path.

4. The method as described in claim 3, characterized in that, Determining the second state of each subpopulation includes: The number of individuals in the subpopulation is determined as the first number, and the number of individuals in the subpopulation whose local optimal fitness decreases is determined as the second number. The local fitness reduction rate of the subpopulation is determined based on the first quantity and the second quantity; The second state of each subpopulation is determined based on the local fitness reduction rate.

5. The method as described in claim 3, characterized in that, The step of determining the new expected reward for an individual in the population to perform the second selection action in the second state, and updating the second Q table based on the new expected reward, further includes: Instant rewards are determined based on the distance between individual populations and the target point; When it is determined that the second state is within the first range and its corresponding local fitness reduction rate decreases, the target reward is determined to be a first fixed value; When it is determined that the second state is within the second range and its corresponding local fitness reduction rate increases, the target reward is determined to be the second fixed value; When it is determined that the local fitness reduction rate corresponding to the second state decreases, the target reward is determined to be a negative immediate reward; When it is determined that the local fitness reduction rate corresponding to the second state increases, the target reward is determined to be a positive immediate reward; Based on the target reward, determine the new expected reward when an individual in the population performs the second choice action in the second state, and update the second Q table according to the new expected reward.

6. An unmanned vehicle path planning device, characterized in that, The unmanned vehicle path planning device includes: An initialization module is used to initialize parameters, wherein the parameters include algorithm parameters and model parameters, the algorithm parameters are relevant parameters of the Q-learning butterfly optimization algorithm, and the model parameters are relevant parameters of the autonomous vehicle path planning model; A module is established to create a terrain map of the autonomous vehicle's travel area and determine the coordinates of obstacles, the starting point coordinates of the autonomous vehicle, and the ending point coordinates of the autonomous vehicle in the terrain map. The establishment module is also used to establish an initial population based on N randomly generated butterfly individuals; The calculation module is used to calculate the path cost of each individual in the initial population based on the model parameters, the obstacle coordinates, the starting coordinates of the unmanned vehicle, and the ending coordinates of the unmanned vehicle, and to determine the globally optimal individual. The determination module is used to determine the new path corresponding to each group of individuals based on the switching probability in the algorithm parameters; The determining module is further configured to determine a new globally optimal individual based on the new path; The judgment module is used to determine whether the current iteration count is equal to the maximum iteration count; The execution module is used to take the new global optimal individual as the output global optimal individual and determine the optimal path for the unmanned vehicle based on the output global optimal individual; The determining module is also used to generate random values ​​for individuals in various groups; Determine whether the random value is less than the switching probability; If so, the paths of individuals in the population are updated using the Q-learning global search mechanism to obtain new paths. The Q-learning global search mechanism is an improvement on the global search process of the BOA algorithm based on the Q-learning mechanism. If not, the path of the individual in the population is updated by using the local development mechanism of Q-learning to optimize the inertia weight, and a new path is obtained. The local development mechanism of Q-learning to optimize the inertia weight is obtained by improving the local development process of the BOA algorithm based on the Q-learning mechanism. The determining module is further configured to, in step S5031, define a first state set and a first action set, wherein the first state set includes a first global state, a second global state and a third global state, and the first action set includes a first global action, a second global action and a third global action. S5032. Based on the first state set and the first action set, generate a first Q table for the population individuals, wherein the first Q table stores the expected reward corresponding to the state-action combination. S5033. Determine the first state and first action of an individual in the population, wherein the first action is an action randomly selected from the first action set with probability p and a first maximum action selected with probability 1-p, the first maximum action is the action corresponding to the maximum expected reward in the first Q table, and the probability p is the switching probability. S5034. Determine the new expected reward when an individual in the population performs the first selection action in the first state, and update the first Q table according to the new expected reward; S5035. Update the path of the population individuals according to the first selection action to obtain the updated path; S5036. Determine whether the first iteration number is equal to the first preset iteration number; S5037. If not, then after re-determining the first state in step S5033 based on the updated path and re-determining the first selection action in step S5033 based on the updated first Q table, repeat steps S5033-S5037; if yes, then use the updated path as the new path.

7. An unmanned vehicle path planning device, characterized in that, The device includes: a memory, a processor, and an autonomous vehicle path planning program stored in the memory and executable on the processor, the autonomous vehicle path planning program being configured to implement the steps of the autonomous vehicle path planning method as described in any one of claims 1 to 5.

8. A storage medium, characterized in that, The storage medium stores an autonomous vehicle path planning program, which, when executed by a processor, implements the steps of the autonomous vehicle path planning method as described in any one of claims 1 to 5.

Images