A path planning method for intelligent robots
Through neural network estimation of action value functions, designing safe areas and using Bezier curve smooth paths, the problems of inaccurate capture of action value functions, low adaptability of path planning and safety in intelligent robot path planning methods are solved, and a higher Path planning accuracy and robustness.
Patent Information
- Application Number
- CN202411098232.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-12
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2044-08-12
AI Technical Summary
The existing intelligent robot path planning method has the inability to accurately capture the complex relationship between state and action, the path planning adaptability is low, the robot's mobile security is low, the reward function is improperly designed, the training process is unstable, and the initial path is unnecessary. The improper setting of turning points and path planning parameters leads to low accuracy.
The neural network estimation action value function is used to design safe and prohibited areas, use Bezier curves to smooth the initial path, and initialize adaptive adjustments of multiple sets of parameters to improve the robustness and adaptability of path planning.
It improves the nonlinear approximation ability and adaptability of path planning, enhances the safety of robot movement and the accuracy of path planning, reduces jitter and control duration during training, and improves the effectiveness and robustness of path planning.
Smart Images

Figure CN118625816B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the technical field of path planning, and in particular to a path planning method for an intelligent robot. Background Art
[0002] Intelligent robot path planning methods refer to methods that use artificial intelligence and machine learning technologies to enable robots to autonomously plan paths in complex environments, and to continuously learn and optimize path planning strategies in the process of interacting with the environment. However, general robot path planning methods have problems such as the action value function cannot accurately capture the complex relationship between state and action, and the path planning has low adaptability; general robot path planning methods have problems such as low safety during robot movement, improper reward function design, and unstable training process; general robot path planning methods have problems such as unnecessary turns in the initial path, and improper setting of path planning parameters leading to low accuracy. Summary of the invention
[0003] In view of the above situation, in order to overcome the defects of the prior art, the present invention provides an intelligent robot path planning method. In view of the problem that the action value function of the general robot path planning method cannot accurately capture the complex relationship between the state and the action, and the path planning has low adaptability, the present solution uses a neural network to estimate the action value function, which has stronger nonlinear approximation ability, stronger adaptability and superior generalization ability; in view of the problem that the general robot path planning method has low safety in robot movement, improper reward function design and unstable training process, the present solution designs a safe area and a prohibited area, and the reward function carefully considers the position and action of the robot, gives different rewards or penalties for different situations, introduces a smoothing term and a maximum number of training times in the reward function, reduces the jitter in the training process and controls the training time; in view of the problem that the initial path of the general robot path planning method has unnecessary turns and the path planning parameters are improperly set, resulting in low accuracy, the present solution uses a Bezier curve to smooth the initial path, reduces the jitter on the path, and improves the effect of path planning; initializes multiple sets of different parameters and adaptively adjusts the parameters, so that the parameter update is more global; thereby making the path planning method more robust and adaptable.
[0004] The technical solution adopted by the present invention is as follows: The present invention provides an intelligent robot path planning method, the method comprising the following steps:
[0005] Step S1: data collection;
[0006] Step S2: data preprocessing;
[0007] Step S3: designing action value function;
[0008] Step S4: design reward function;
[0009] Step S5: path planning.
[0010] Furthermore, in step S1, the data collection is to collect robot motion data; the robot motion data is the data of robot motion recorded in different environments, including state, action, environmental information, obstacle position, target position, actual path and reward obtained.
[0011] Furthermore, in step S2, the data preprocessing is to clean, normalize and label the collected data; the data cleaning is to remove noise and erroneous data, and process missing values; the normalization is to normalize the data; and the data labeling is to ensure that each data point contains the state, the action taken and the result.
[0012] Further, in step S3, the design of the action value function is to construct a Q-learn network structure based on a neural network; specifically, the following steps are included:
[0013] Step S31: network structure design, the established neural network is a three-layer static forward network, including an input layer, a hidden layer and an output layer;
[0014] Step S32: Input layer design. The input layer consists of signal source nodes, and its input consists of state variables and an action variable a; the number of inputs is M=N+1, and the input vector is expressed as: X=[ s 1 , s 2 ,…, s N ,a] ; Among them, s1, s2 and s N are the 1st, 2nd and Nth state variables respectively; N is the total number of state variables; X is the input vector; M is the number of inputs;
[0015] Step S33: hidden layer design, the hidden layer uses radial basis function; it is expressed as follows:
[0016] ;
[0017] ;
[0018] ;
[0019] In the formula, is the response value of the input vector X at the kth radial basis function node; μ i and σ iare the Gaussian function cluster center and variance corresponding to the i-th hidden neuron; μ M and σ M are the Gaussian function cluster center and variance corresponding to the Mth hidden neuron; max It is the maximum distance that the robot can travel between the current position and the target point position;
[0020] Step S34: Output layer design. The output layer is used to estimate the action value function and contains only one output node. The action value function is expressed as follows:
[0021] ;
[0022] Where Q(·) is the action value function; s t and a t are the state and selected action at time step t respectively; de is the distance between the current position and the target position of the robot.
[0023] Furthermore, in step S4, the design reward function specifically includes the following steps:
[0024] Step S41: Design a safe area and introduce a safety threshold ρ; it is expressed as follows:
[0025] ;
[0026] ;
[0027] Where L is the length of the robot; x0 and y0 are the horizontal and vertical center coordinates of the safety zone; x and y are the horizontal and vertical coordinates of the robot; r is the radius of the zone
[0028] Step S42: Design the forbidden area; it is expressed as follows:
[0029] ;
[0030] Where y(t) is the vertical coordinate of the robot’s current position; n1 (t), y n2 (t), y n3 (t) and y n4 (t) are the ordinates of the first boundary line, the second boundary line, the third boundary line and the fourth boundary line at time t;
[0031] Step S43: Design a reward function; when the robot's position is an obstacle, a negative reward will be obtained; when the robot reaches the target point, a large positive reward will be obtained; when the robot is close to the target point, 1 will be obtained as a reward; when the robot stays in place, -1 will be obtained as a reward; in other cases, -2 will be obtained as a reward; the formula used is as follows:
[0032] ;
[0033] ;
[0034] Where d(t) and d(t-1) represent the distance between the robot and the target point at the current time step and the previous time step respectively; d o Indicates the safe distance of the obstacle. If the distance between the robot and the target point is less than do, it means that an obstacle has been encountered; d g Indicates the threshold of the distance to the target point. If the distance between the robot and the target point is less than d g , it is considered to have reached the target point; d n Indicates the distance to the target point. When this range is reached, a positive reward will be given; is the pre-reward; R is the final reward; r a is the weight value; n o (P0-P1) is the normalized distance between the next position P0 of the robot obtained based on the artificial potential field method and the actual position P1 of the robot obtained when the environment is updated; T is the maximum number of training times; is a smoothing term.
[0035] Furthermore, in step S5, the path planning specifically includes the following steps:
[0036] Step S51: smoothing the initial path, smoothing the initial path based on the Bezier curve, replacing the points on the original path with the points on the calculated Bezier curve as new path points; the formula used is as follows:
[0037] ;
[0038] In the formula, , , and represents the initial control point; B(t1) is the Bezier curve equation; t1 is the curve parameter;
[0039] Step S52: path planning, presetting the accuracy threshold; initializing parameters, and setting different sets of parameters; smoothing the initial path based on step S51; based on the Q-learn network designed in step S3 and the reward function designed in step S4; using Q-learning to select the optimal action to achieve path planning; when the path planning accuracy achieved by the robot is higher than the accuracy threshold, the path planning is completed; when the path planning accuracy achieved by the robot is not higher than the accuracy threshold, the parameters are updated; when the maximum number of training times is reached, the parameters are reinitialized; the formula used to update the parameters is as follows:
[0040] ;
[0041] In the formula, and After and before the update The position of the dth parameter of the group; is the learning rate; and They are The accuracy of group parameters and Average accuracy of group parameters; XBd, and They are the optimal group parameters, random group parameters and The position of the empirically optimal group of parameters; is the movement weight.
[0042] The beneficial effects achieved by the present invention using the above scheme are as follows:
[0043] (1) In view of the problem that the action value function of general robot path planning methods cannot accurately capture the complex relationship between state and action, and the path planning adaptability is low, this scheme uses a neural network to estimate the action value function, which has stronger nonlinear approximation ability, stronger adaptability and superior generalization ability.
[0044] (2) In response to the problems of low safety during robot movement, improper reward function design and unstable training process in general robot path planning methods, this scheme designs safe areas and prohibited areas. The reward function carefully considers the position and movement of the robot, and gives different rewards or penalties for different situations. Smoothness terms and maximum number of training times are introduced into the reward function to reduce jitter during training and control the duration of training.
[0045] (3) In view of the problems that the initial path of general robot path planning methods has unnecessary turns and the path planning parameters are improperly set, resulting in low accuracy, this scheme uses Bezier curves to smooth the initial path, reduces the jitter on the path, and improves the effect of path planning; initializes multiple sets of different parameters and adaptively adjusts the parameters, making the parameter update more global; thus making the path planning method more robust and adaptable. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of a flow chart of an intelligent robot path planning method provided by the present invention;
[0047] Figure 2 is a schematic flow chart of step S3;
[0048] Figure 3 It is a schematic diagram of the process of step S4.
[0049] The accompanying drawings are used to provide further understanding of the present invention and constitute a part of the specification. They are used to explain the present invention together with the embodiments of the present invention and do not constitute a limitation of the present invention. DETAILED DESCRIPTION
[0050] The technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments; based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.
[0051] In the description of the present invention, it should be understood that terms such as “upper”, “lower”, “front”, “back”, “left”, “right”, “top”, “bottom”, “inside” and “outside” indicating directions or positional relationships are based on the directions or positional relationships shown in the accompanying drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific direction, be constructed and operated in a specific direction, and therefore should not be understood as limiting the present invention.
[0052] Example 1, see Figure 1 The present invention provides an intelligent robot path planning method, which comprises the following steps:
[0053] Step S1: data collection, collecting robot motion data;
[0054] Step S2: data preprocessing, data cleaning, normalization and data labeling of the collected data;
[0055] Step S3: designing an action value function and using a neural network to estimate the action value function;
[0056] Step S4: Design a reward function, design a safe area and a prohibited area, and introduce a smoothing term and a maximum number of training times into the reward function;
[0057] Step S5: Path planning, using Bezier curves to smooth the initial path, initializing multiple sets of different parameters and adaptively adjusting the parameters, and finally realizing robot path planning.
[0058] Example 2, see Figure 1 ,This embodiment is based on the above embodiment. In step S1, the robot motion data is the data of the robot motion recorded in different environments, including state, action, environmental information, obstacle position, target position, actual path, and reward obtained.
[0059] Example 3, see Figure 1This embodiment is based on the above embodiment. In step S2, data cleaning is to remove noise and erroneous data and process missing values; normalization is to normalize the data; and data labeling is to ensure that each data point contains the state, the action taken, and the result.
[0060] Example 4, see Figure 1 and Figure 2 This embodiment is based on the above embodiment. In step S3, the action value function is designed to construct a Q-learn network structure based on a neural network. Specifically, the following steps are included:
[0061] Step S31: network structure design, the established neural network is a three-layer static forward network, including an input layer, a hidden layer and an output layer;
[0062] Step S32: Input layer design. The input layer consists of signal source nodes, and its input consists of state variables and an action variable a; the number of inputs is M=N+1, and the input vector is expressed as: X=[ s 1 , s 2 ,…, s N ,a] ; Among them, s1, s2 and s N are the 1st, 2nd and Nth state variables respectively; N is the total number of state variables; X is the input vector; M is the number of inputs;
[0063] Step S33: hidden layer design, the hidden layer uses radial basis function; it is expressed as follows:
[0064] ;
[0065] ;
[0066] ;
[0067] In the formula, is the response value of the input vector X at the kth radial basis function node; μ i and σ i are the Gaussian function cluster center and variance corresponding to the i-th hidden neuron; μ M and σ M are the Gaussian function cluster center and variance corresponding to the Mth hidden neuron; max It is the maximum distance that the robot can travel between the current position and the target point position;
[0068] Step S34: Output layer design. The output layer is used to estimate the action value function and contains only one output node. The action value function is expressed as follows:
[0069] ;
[0070] Where Q(·) is the action value function; s t and a t are the state and selected action at time step t respectively; de is the distance between the current position and the target position of the robot.
[0071] By performing the above operations, in order to solve the problem that the action value function of the general robot path planning method cannot accurately capture the complex relationship between state and action, and the path planning adaptability is low, this scheme uses a neural network to estimate the action value function, which has stronger nonlinear approximation ability, stronger adaptability and superior generalization ability.
[0072] Example 5, see Figure 1 and Figure 3 This embodiment is based on the above embodiment. In step S4, designing the reward function specifically includes the following steps:
[0073] Step S41: Design a safe area and introduce a safety threshold ρ; it is expressed as follows:
[0074] ;
[0075] ;
[0076] Where L is the length of the robot; x0 and y0 are the horizontal and vertical center coordinates of the safety zone; x and y are the horizontal and vertical coordinates of the robot; r is the radius of the zone
[0077] Step S42: Design the forbidden area; it is expressed as follows:
[0078] ;
[0079] Where y(t) is the vertical coordinate of the robot’s current position; n1 (t), y n2 (t), y n3 (t) and y n4 (t) are the ordinates of the first boundary line, the second boundary line, the third boundary line and the fourth boundary line at time t;
[0080] Step S43: Design a reward function; when the robot's position is an obstacle, a negative reward will be obtained; when the robot reaches the target point, a large positive reward will be obtained; when the robot is close to the target point, 1 will be obtained as a reward; when the robot stays in place, -1 will be obtained as a reward; in other cases, -2 will be obtained as a reward; the formula used is as follows:
[0081] ;
[0082] ;
[0083] Where d(t) and d(t-1) represent the distance between the robot and the target point at the current time step and the previous time step respectively; d o Indicates the safe distance of the obstacle. If the distance between the robot and the target point is less than do, it means that an obstacle has been encountered; d g Indicates the threshold of the distance to the target point. If the distance between the robot and the target point is less than d g , it is considered to have reached the target point; d n Indicates the distance to the target point. When this range is reached, a positive reward will be given; is the pre-reward; R is the final reward; r a is the weight value; n o (P0-P1) is the normalized distance between the next position P0 of the robot obtained based on the artificial potential field method and the actual position P1 of the robot obtained when the environment is updated; T is the maximum number of training times; is a smoothing term.
[0084] By performing the above operations, this solution designs safe areas and prohibited areas to address the problems of low safety in robot movement, improper reward function design and unstable training process in general robot path planning methods. The reward function carefully considers the position and movement of the robot, and gives different rewards or penalties for different situations. Smoothness terms and maximum training times are introduced into the reward function to reduce jitter during training and control the duration of training.
[0085] Example 6, see Figure 1 This embodiment is based on the above embodiment. In step S5, the path planning specifically includes the following steps:
[0086] Step S51: smoothing the initial path, smoothing the initial path based on the Bezier curve, replacing the points on the original path with the points on the calculated Bezier curve as new path points; the formula used is as follows:
[0087] ;
[0088] In the formula, , , and represents the initial control point; B(t1) is the Bezier curve equation; t1 is the curve parameter;
[0089] Step S52: path planning, presetting the accuracy threshold; initializing parameters, and setting different sets of parameters; smoothing the initial path based on step S51; based on the Q-learn network designed in step S3 and the reward function designed in step S4; using Q-learning to select the optimal action to achieve path planning; when the path planning accuracy achieved by the robot is higher than the accuracy threshold, the path planning is completed; when the path planning accuracy achieved by the robot is not higher than the accuracy threshold, the parameters are updated; when the maximum number of training times is reached, the parameters are reinitialized; the formula used to update the parameters is as follows:
[0090] ;
[0091] In the formula, and After and before the update The position of the dth parameter of the group; is the learning rate; and They are The accuracy of group parameters and Average accuracy of group parameters; XBd, and They are the optimal group parameters, random group parameters and The position of the empirically optimal group of parameters; is the movement weight.
[0092] By performing the above operations, in order to solve the problems of unnecessary turns in the initial path and low accuracy caused by improper setting of path planning parameters in general robot path planning methods, this solution uses Bezier curves to smooth the initial path, reduces the jitter on the path, and improves the effect of path planning; initializes multiple sets of different parameters and adaptively adjusts the parameters to make the parameter update more global; thereby making the path planning method more robust and adaptable.
[0093] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.
[0094] While the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that many changes, modifications, substitutions and variations can be made to the embodiments without departing from the principles and spirit of the invention.
[0095] The present invention and its embodiments are described above, and such description is not restrictive. The drawings show only one embodiment of the present invention, and the actual structure is not limited thereto. In short, if ordinary technicians in the field are inspired by it, without departing from the purpose of the invention, they can design a structure and embodiment similar to the technical solution without creativity, which should belong to the protection scope of the present invention.
Claims
1. A path planning method for an intelligent robot, characterized in that: The method comprises the following steps: Step S1: data collection; Step S2: data preprocessing; Step S3: designing action value function; Step S4: Design a reward function, design a safe area and a prohibited area, and introduce a smoothing term and a maximum number of training times into the reward function; Step S5: path planning, using Bezier curves to smooth the initial path, initializing multiple sets of different parameters and adaptively adjusting the parameters, and finally realizing robot path planning; In step S4, step S43 is included: designing a reward function; when the robot's position is an obstacle, a negative reward will be obtained; when the robot reaches the target point, a large positive reward will be obtained; when the robot is close to the target point, 1 will be obtained as a reward; when the robot stays in place, -1 will be obtained as a reward; in other cases, -2 will be obtained as a reward; the formula used is as follows: ; ; Where d(t) and d(t-1) represent the distance between the robot and the target point at the current time step and the previous time step respectively; d o Indicates the safe distance of the obstacle. If the distance between the robot and the target point is less than d o , it means that an obstacle has been encountered; d g Indicates the threshold of the distance to the target point. If the distance between the robot and the target point is less than d g , it is considered to have reached the target point; d n Indicates the distance to the target point. When this range is reached, a positive reward will be given; is the pre-reward; R is the final reward; r a is the weight value; n o (P0-P1) is the normalized distance between the next position P0 of the robot obtained based on the artificial potential field method and the actual position P1 of the robot obtained when the environment is updated; T is the maximum number of training times; is a smoothing term; In step S3, the design of the action value function is to construct a Q-learn network structure based on a neural network; specifically, the following steps are included: Step S31: network structure design, the established neural network is a three-layer static forward network, including an input layer, a hidden layer and an output layer; Step S32: Input layer design. The input layer consists of signal source nodes, and its input consists of state variables and an action variable a; the number of inputs is M=N+1, and the input vector is expressed as: ; Among them, s1, s2 and s N are the 1st, 2nd and Nth state variables respectively; N is the total number of state variables; X is the input vector; M is the number of inputs; Step S33: hidden layer design, the hidden layer uses radial basis function; it is expressed as follows: ; ; ; In the formula, is the response value of the input vector X at the kth radial basis function node; μ i and σ i are the Gaussian function cluster center and variance corresponding to the i-th hidden neuron; μ M and σ M are the Gaussian function cluster center and variance corresponding to the Mth hidden neuron; max It is the maximum distance that the robot can travel between the current position and the target point position; Step S34: Output layer design. The output layer is used to approximate the action value function and contains only one output node. The action value function is expressed as follows: ; Where Q(·) is the action value function; s t and a t are the state and selected action at time step t respectively; de is the distance between the current position and the target position of the robot; In step S5, the path planning specifically includes the following steps: Step S51: smoothing the initial path, smoothing the initial path based on the Bezier curve, replacing the points on the original path with the points on the calculated Bezier curve as new path points; the formula used is as follows: ; In the formula, , , and represents the initial control point; B(t1) is the Bezier curve equation; t1 is the curve parameter; Step S52: path planning, presetting the accuracy threshold; initializing parameters, and setting different sets of parameters; smoothing the initial path based on step S51; based on the Q-learn network designed in step S3 and the reward function designed in step S4; using Q-learning to select the optimal action to achieve path planning; when the path planning accuracy achieved by the robot is higher than the accuracy threshold, the path planning is completed; when the path planning accuracy achieved by the robot is not higher than the accuracy threshold, the parameters are updated; when the maximum number of training times is reached, the parameters are reinitialized; the formula used to update the parameters is as follows: ; In the formula, and After and before the update The position of the dth parameter of the group; is the learning rate; and They are The accuracy of group parameters and Average accuracy of group parameters; XB d , and They are the optimal group parameters, random group parameters and The position of the empirically optimal group of parameters; is the movement weight; In step S4, the design reward function specifically includes the following steps: Step S41: Design a safe area and introduce a safety threshold ρ; it is expressed as follows: ; ; Where L is the length of the robot; x0 and y0 are the horizontal and vertical center coordinates of the safety zone; x and y are the horizontal and vertical coordinates of the robot; r is the radius of the zone Step S42: Design the forbidden area; it is expressed as follows: ; Where y(t) is the vertical coordinate of the robot’s current position; n1 (t), y n2 (t), y n3 (t) and y n4 (t) are the ordinates of the first boundary line, the second boundary line, the third boundary line and the fourth boundary line at time t; Step S43: Design a reward function.
2. The intelligent robot path planning method according to claim 1, characterized in that: In step S1, the data collection is to collect robot motion data; the robot motion data is the data of robot motion recorded in different environments, including state, action, environmental information, obstacle position, target position, actual path and reward obtained.
3. The intelligent robot path planning method according to claim 1, characterized in that: In step S2, the data preprocessing is to clean, normalize and label the collected data; the data cleaning is to remove noise and erroneous data and process missing values; the normalization is to normalize the data; and the data labeling is to ensure that each data point contains the state, action taken and result.
