A composite obstacle avoidance control method and device based on reinforcement learning
By combining a linear convex combination of MPI and SFA with a behavior-evaluation neural network, the optimality problem of local obstacle avoidance methods on the motion trajectory is solved, realizing safe and efficient obstacle avoidance of autonomous unmanned systems and generating smooth motion trajectories.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- TIANJIN UNIV
- Filing Date
- 2022-06-23
- Publication Date
- 2026-04-17
AI Technical Summary
Existing local obstacle avoidance methods fail to achieve optimality across the entire trajectory, and the reward settings are unclear, making it impossible to balance local safety with data utilization during autonomous exploration.
A composite obstacle avoidance control method based on reinforcement learning is adopted, which combines the Model-Policy Iteration (MPI) method and the State-Following Avoidance (SFA) method. An obstacle avoidance control model is constructed through linear convex combination, and switching and scheduling functions are designed using behavior-evaluation neural networks and reinforcement learning to achieve safe obstacle avoidance.
While ensuring regional optimization performance, it has achieved safety obstacle avoidance for autonomous unmanned systems, with shorter movement trajectories, higher safety, and better optimization.
Smart Images
Figure CN115933630B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the fields of reinforcement learning and optimal control, and more specifically, to a composite obstacle avoidance control method and apparatus based on reinforcement learning. Background Technology
[0002] In recent years, artificial intelligence technologies, represented by reinforcement learning, have greatly promoted the development and application of autonomous unmanned systems, such as driverless cars and drones. These motion systems typically face complex obstacle environments when performing tasks, and reliable obstacle avoidance control methods are needed to successfully complete these tasks. To achieve rapid online decision-making, local obstacle avoidance methods, such as the artificial potential field method, have received widespread attention.
[0003] Local obstacle avoidance methods only consider safe obstacle avoidance as the control objective, without taking into account the optimality of the control strategy over the entire trajectory. Therefore, their optimization capabilities need further improvement. Thus, reinforcement learning-based intelligent control methods offer a solution to this problem by utilizing neural networks and policy iteration to achieve an approximate solution for the control strategy.
[0004] Although many studies have proposed obstacle avoidance control schemes based on reinforcement learning, these methods still have some problems, such as unclear reward settings, inability to balance local safety in policy optimization, and low data utilization in autonomous exploration. Therefore, there is an urgent need to develop a novel obstacle avoidance control method to meet practical applications. Summary of the Invention
[0005] This invention provides a composite obstacle avoidance control method and apparatus based on reinforcement learning to solve the obstacle avoidance problem of autonomous unmanned systems. The method includes a linear convex composite framework that balances regional optimality and local obstacle avoidance, and obtains an obstacle avoidance control strategy based on reinforcement learning using an Action-Evaluation Network (ACNN) structure. The specific technical solution is as follows:
[0006] A composite obstacle avoidance control method based on reinforcement learning, the method comprising the following steps:
[0007] S10. Establish an obstacle avoidance environment and an optimal control model for obstacle avoidance;
[0008] S20. Based on the obstacle environment and the optimal control model, a composite obstacle avoidance control model is constructed by switching between a linear convex combination of model-based policy iteration (MPI) and state-following avoidance (SFA); the composite obstacle avoidance control model is as follows:
[0009]
[0010] Where λ(x) is the switching function, and its value also changes smoothly between 0 and 1; It is the estimate of the cost function by the MPI method; It is the estimate of the cost function by the SFA method;
[0011] S30. Construct an obstacle avoidance control strategy based on behavior-evaluation neural networks and reinforcement learning;
[0012] S40. Use extrapolation to obtain empirical data and determine adaptive network update rules.
[0013] Further: The process of constructing the obstacle avoidance environment and the optimal control model for obstacle avoidance in step S10:
[0014] Step S101: Construct an obstacle environment around the obstacle object; that is:
[0015] The intelligent agent of the autonomous unmanned system is characterized by the following affine nonlinear model:
[0016]
[0017] Where: f(x(t)) is the internal dynamic information, g(x(t)) represents the control coupling matrix, and u(t) represents the control input or control strategy;
[0018] The intelligent agent uses an obstacle detector to detect the real-time distance to each obstacle. Its control objective is to safely bypass every obstacle and reach the target point x. e The number of obstacles is denoted as N. o ;
[0019] The obstacle avoidance environment treats obstacles as circular or spherical areas, establishing corresponding obstacle areas O. i Based on this, conflict zone C is radiated out. i Avoidance area and switch regions
[0020] The area faced by an intelligent agent in obstacle avoidance can be represented as
[0021] Step S102: Construct a safe operating domain and design barrier functions;
[0022] Using forward invariance, the obstacle region can be represented as
[0023] Where h(x) is a smooth function, and the safe operating domain of the system is S = Ψ - 0, where Ψ represents the operating space of the system;
[0024] Based on the safe operating domain, the following barrier function B(x) is designed using the barrier function method:
[0025]
[0026] Where: K b It is a positive gain, and s(x) is a smoothly changing scheduling function whose value changes smoothly between 0 and 1.
[0027] Step S103: Establish an optimal control model based on the cost function;
[0028] System operating cost x T Qx, controlling energy costs u T Given Ru and the barrier function B(x), the following cost function to be optimized is established:
[0029]
[0030] Where Q and R are positive definite and symmetric constant matrices used to measure the size of specific cost items;
[0031] Using the Bellman optimality principle, the optimal control model for obstacle avoidance is established as follows:
[0032]
[0033]
[0034] Among them: ▽J * It is the optimal cost function J * (x) is the partial derivative of (x) with respect to the state x.
[0035] Furthermore, the rules for the linear convex combination of the policy iteration method and the state following avoidance method in step 20 are as follows:
[0036] The strategy iteration method implements the switching function through the following formula, which affects when the strategy iteration method starts to switch to the state-following avoidance method.
[0037]
[0038] Where: λ(x) contains N o There are 1 component, each smoothly varying between 0 and 1. If obstacle i is never within the detection range of the agent, then the corresponding component λ... i (x) will never be activated, i.e., λ i (x) = 0;
[0039] The state-following avoidance method implements the scheduling function through the following formula, which affects the obstacle avoidance process of the optimal control model;
[0040]
[0041] Wherein, s(x) also contains N o There are 1 component, each of which smoothly varies between 0 and 1; if the component corresponding to obstacle i is λ i If (x) is not activated, then s i (x) is also never activated; as the agent approaches the obstacle region, the scheduling function will approach 1, which means that the obstacle function has a greater impact on the cost function. This will conflict with the policy optimization behavior, and thus cause the agent to take avoidance behavior in subsequent actions.
[0042] Furthermore, the process of constructing the behavior-evaluation neural network and the obstacle avoidance control strategy through reinforcement learning in step S30 is as follows:
[0043] Step S301: Design an obstacle penalty term B based on the obstacle function B(x). o (x) is used to characterize the reward and punishment situation of the intelligent agent in obstacle avoidance; on this basis, an evaluation network and a behavior network are constructed using a multi-layer feedforward neural network.
[0044] ACNN architectures are designed for both the policy iteration method and the state-following avoidance method. The agent comprises two evaluation networks and two behavior networks; that is:
[0045] The evaluation network for the policy iteration method is constructed as follows
[0046] Among them: is The evaluation weights are σ(x), which is the corresponding activation function.
[0047] The evaluation network construction of the state-following avoidance method is as follows
[0048] in: It is an evaluation weight. Here, c(x) is the state-following basis function, and c(x) is the corresponding kernel function. Based on this, through a linear convex combination of the following formulas, the approximate value of the cost function is obtained as follows:
[0049]
[0050] In the above formula, λ(x) is simplified to λ; for behavioral networks, a similar structure is adopted, with its weights... Used to estimate the evaluation weight w c1 ,w c2 That is: obstacle avoidance control strategy based on reinforcement learning
[0051]
[0052] Among them: the obstacle avoidance control strategy includes the weight parameters of two behavioral networks;
[0053] Step S302: Apply the Bellman equation to the cost function. and obstacle avoidance control strategies The real-time Bellman error δ(t) is calculated as follows:
[0054]
[0055] Among them, Bellman error includes all neural network parameters and represents the learning error in reinforcement learning.
[0056] Furthermore, in step S40, extrapolation is used to obtain empirical data and determine the adaptive network update rules:
[0057] Step S401: Obtain empirical data using the state extrapolation method;
[0058] The trajectory points generated by extrapolation using the strategy iteration method are: The trajectory points generated by the state-following avoidance method extrapolation are: Computational extrapolation control strategy
[0059]
[0060]
[0061] The trajectory points generated by extrapolation using the slight iterative method are: The trajectory points generated by the state-following avoidance method extrapolation are: Calculate the extrapolation Bellman error δ 1,k (t),δ 2,j (t):
[0062]
[0063]
[0064] Step S402: Determine the adaptive update rules for the evaluation and behavior networks using the gradient descent adaptive method and the gradient projection operator method; that is:
[0065] The adaptive update rule for the evaluation network obtained using the gradient descent adaptive method is as follows:
[0066]
[0067]
[0068] Where: k c11 ,kc12 >0 is the evaluation network learning rate for the MPI method, k c21 ,k c22 >0 is the evaluation network learning rate for the SFA method;
[0069] The adaptive update rule for the behavioral network obtained using the gradient projection operator method is as follows:
[0070]
[0071]
[0072] Where: proj{·} is the projection operator, k a1 >0 represents the learning rate of the behavior network in the MPI method, k a2 >0 is the learning rate of the behavioral network in the SFA method, F a1 ,F a2 It is a positive definite constant matrix;
[0073] Step S403: Composite obstacle avoidance is achieved through reinforcement learning; wherein the evaluation network and the behavior network perform policy evaluation and policy improvement respectively, and under the action of the switching function, the two pairs of ACNN structures are switched reasonably to generate continuous cost estimates and control outputs.
[0074] The present invention also includes a technical solution for a composite obstacle avoidance control device based on reinforcement learning, the device comprising:
[0075] An obstacle detector is used to detect the distance between an unmanned system and obstacles and feeds this information back to the microprocessor; however, this detector has a limited detection range, i.e., a finite detection radius.
[0076] The microprocessor, mainly consisting of embedded chips and related functional circuits, is used to execute any one of the steps of claims 1-5, and its output signal is the obstacle avoidance control strategy obtained by behavior network calculation.
[0077] The memory, interconnected with the microprocessor, can save the motion trajectory data of the unmanned system, save the learning data generated by the microprocessor, and also periodically transmit the stored experience data to the microprocessor.
[0078] The drive component is responsible for executing the control commands output by the microprocessor, thereby providing power to the unmanned system.
[0079] Beneficial effects
[0080] The beneficial effects of the technical solution provided by this invention are:
[0081] This invention addresses the obstacle avoidance problem in autonomous unmanned systems. Utilizing optimal control theory and reinforcement learning techniques, it proposes a composite obstacle avoidance control method based on reinforcement learning. This method achieves obstacle avoidance while ensuring optimal regional performance, aligning with the application needs and development trends of related technologies. A search of existing literature and technologies revealed no similar technical solutions.
[0082] This invention combines the advantages of both MPI and SFA methods, utilizing reinforcement learning to achieve online solution of obstacle avoidance control strategies, enabling the agent to safely reach the target point in multi-obstacle environments. Compared with existing methods, it achieves shorter trajectories, higher safety, and better optimization. Attached Figure Description
[0083] Figure 1 This is a flowchart illustrating a composite obstacle avoidance control method based on reinforcement learning according to one embodiment;
[0084] Figure 2 This is a flowchart illustrating step S10 according to one embodiment;
[0085] Figure 3 This is a flowchart illustrating step S20 according to one embodiment;
[0086] Figure 4 This is a flowchart illustrating step S30 according to one embodiment;
[0087] Figure 5 This is a flowchart illustrating step S40 according to one embodiment;
[0088] Figure 6 This is a safety trajectory diagram illustrating the obstacle avoidance process of an autonomous vehicle system according to one embodiment;
[0089] Figure 7 The following describes the change process of the switching function and scheduling function according to one embodiment;
[0090] Figure 8 This is a comparison diagram of the safe trajectory of an unmanned vehicle system during obstacle avoidance, according to one embodiment.
[0091] Figure 9 This is a block diagram of a reinforcement learning-based composite obstacle avoidance control device according to one embodiment. Detailed Implementation
[0092] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be described in further detail below. It should be noted that, unless otherwise stated, the steps, mathematical expressions, and numerical values set forth in these embodiments do not limit the scope of this disclosure. Furthermore, it should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not drawn to actual scale.
[0093] like Figure 1-9 As shown, this invention provides a composite obstacle avoidance control method based on reinforcement learning, the method comprising:
[0094] First, an obstacle avoidance environment and an optimal control model for obstacle avoidance are constructed. Due to the integrated control decision-making and inherent complex nonlinearity of autonomous unmanned systems, they are treated as intelligent agents represented by an affine nonlinear model. This agent possesses local detection capabilities, enabling it to detect the distance to obstacles using obstacle detectors. Its control objective is to safely bypass obstacles and reach the target point. An obstacle environment is constructed around the obstacles, treating them as circular or spherical regions, and establishing corresponding obstacle regions. Conflict regions, avoidance regions, and switching regions are then derived from this. Based on the obstacle environment, a safe operating domain is defined using forward invariance, and an obstacle function is designed, which includes a scheduling function. Considering the system operating cost, control energy cost, and obstacle function, a cost function is designed. Based on the cost function, the optimal control model for the obstacle avoidance problem is established using the Bellman optimality principle.
[0095] Alternatively, the switching region can also be described as a detection region, where the agent will detect obstacles upon entering. In this disclosure, the switching region involves switching between two methods: when the agent enters the switching region, the Model-Policy Iteration (MPI) method gradually switches to the State-Following Avoidance (SFA) method; when the agent moves into the avoidance region, it fully switches to SFA. The process is reversed when the agent moves away from the obstacle.
[0096] Furthermore, switching rules are designed to achieve composite obstacle avoidance control. For the obstacle environment and optimal control model, a general framework for the composite obstacle avoidance control method is constructed using a linear convex combination of MPI and SFA methods. This framework outlines how to calculate the cost function under the combined action of the two methods. Switching rules for the MPI method are designed, primarily consisting of a switching function to control when the MPI method switches to SFA. Switching rules for the SFA method are also designed, primarily consisting of the aforementioned scheduling function to control when the obstacle function is introduced into the cost function.
[0097] Optionally, both the switching function and the scheduling function can vary smoothly between 0 and 1, and can be selected as sine or cosine functions or hyperbolic tangent functions.
[0098] Furthermore, an obstacle avoidance control strategy based on a behavior-evaluation neural network and reinforcement learning is constructed. An obstacle penalty term is designed based on the obstacle function, and then an evaluation network and a behavior network are constructed using a multi-layer feedforward neural network. Based on the constructed ACNN structure, a reinforcement learning-based obstacle avoidance control strategy is obtained. This control strategy is the actual output of the controller, combining optimization and obstacle avoidance functions. The obtained cost function and obstacle avoidance control strategy are then fed into the optimal control model to calculate the real-time Bellman error (or learning error).
[0099] Alternatively, the neural network described above can be constructed using typical nonlinear polynomial basis functions or state-following basis functions.
[0100] Furthermore, extrapolation is used to obtain empirical data and determine adaptive network update rules. To avoid potential exploration hazards, a state extrapolation method is used to obtain empirical data, i.e., virtual exploration is performed around the neighborhood of the current state, and the empirical data points are the extrapolated trajectories of the current state. Extrapolation control policies and extrapolation Bellman errors for both methods are calculated at the extrapolated trajectory points. Adaptive update rules for the evaluation and behavior networks are determined using the gradient descent adaptive method and the gradient projection operator method. Then, a composite obstacle avoidance process is implemented through reinforcement learning, where the evaluation network and behavior network perform policy evaluation and policy improvement, respectively.
[0101] Optionally, state extrapolation can be performed by adding an extrapolation vector to the current state vector, and the extrapolation vector can be randomly generated using a uniform distribution, normal distribution, or other methods.
[0102] The present invention also provides a composite obstacle avoidance control device based on reinforcement learning, mainly comprising:
[0103] An obstacle detector is used to detect the distance between an unmanned system and an obstacle and feed it back to a microprocessor. The detector has a limited detection range, i.e. a limited detection radius. In relevant embodiments, the obstacle detector may be a lidar, etc.
[0104] Microprocessors, mainly embedded chips (such as ARM) and related functional circuits, are used to perform neural network training and control calculations. Their output signals are the obstacle avoidance control strategies obtained by behavioral network calculations.
[0105] The memory, interconnected with the microprocessor, can save the motion trajectory data of the unmanned system, save the learning data generated by the microprocessor, and also periodically transmit the stored experience data to the microprocessor.
[0106] The drive component is responsible for executing the control commands output by the microprocessor, thereby providing power to the unmanned system. The autonomous unmanned system contemplated in this disclosure has a symmetrical drive component, which, in related embodiments, can be a micro-motor device.
[0107] Example 1
[0108] For autonomous unmanned systems with local detection capabilities, this invention discloses a composite obstacle avoidance control method based on reinforcement learning. (See also...) Figure 1 The method includes the following steps:
[0109] S10: Construct the obstacle avoidance environment and the optimal control model for obstacle avoidance;
[0110] Considering the integrated control and decision-making processes and inherent complex nonlinearities of autonomous unmanned systems, an affine nonlinear model can be used in this embodiment. This characterizes the intelligent agent; the agent possesses local detection capabilities and can utilize obstacle detectors to detect the real-time distance to each obstacle. Its control objective is to safely bypass every obstacle and reach the target point x. e The number of obstacles is denoted as N. o Construct an obstacle environment around the obstacle, treating the obstacle as a circular or spherical area, and establish the corresponding obstacle region O. i Based on this, conflict zone C is radiated out. i Avoidance area and switch regions Based on the obstacle environment, a safe operating domain S is defined using forward invariance, and then an obstacle function B(x) is designed. This obstacle function includes a scheduling function s(x), whose value smoothly varies between 0 and 1. Taking into account the system operating cost, control energy cost, and obstacle function, a cost function J(x) is designed. Based on the cost function, an optimal control model for the obstacle avoidance problem is established using the Bellman optimality principle.
[0111] S20: Design method switching rules to achieve composite obstacle avoidance control;
[0112] Specifically, for the obstacle environment and optimal control model, a linear convex combination of the MPI and SFA methods is used to construct the overall framework of the composite obstacle avoidance control method. In this embodiment, it is implemented in the following manner. The cost function is calculated, where λ(x) is the scheduling function, and its value also smoothly varies between 0 and 1. The switching rule for the MPI method is designed, which is mainly composed of the switching function λ(x). In this embodiment, a cosine function is used to implement the switching between MPI and SFA. The switching rule for the SFA method is also designed, which is mainly composed of the scheduling function s(x). Similarly, a cosine function is used to adjust the role of the barrier function in the cost function.
[0113] S30: Construct an obstacle avoidance control strategy based on behavior-evaluation neural networks and reinforcement learning;
[0114] Furthermore, an obstacle avoidance control strategy based on behavior-evaluation neural networks and reinforcement learning is constructed. An obstacle penalty term B is designed based on the obstacle function B(x). o (x) is used to characterize the reward and punishment situation of the agent's obstacle avoidance. Based on this, an evaluation network and a behavior network are constructed using a multi-layer feedforward neural network. It is important to note that ACNN structures need to be designed separately for MPI and SFA, resulting in an agent containing two evaluation networks and two behavior networks. Based on the constructed ACNN structure, a reinforcement learning-based obstacle avoidance control strategy is obtained. This control strategy is the actual output of the controller, combining optimization and obstacle avoidance functions. The obtained cost function... and obstacle avoidance control strategies Substitute the data into the optimal control model to calculate the real-time Bellman error δ(t) (or learning error).
[0115] S40: Use extrapolation to obtain empirical data and determine adaptive network update rules;
[0116] Furthermore, extrapolation is used to obtain empirical data and determine the adaptive network update rules. To avoid potential exploration risks, a state extrapolation method is used to obtain empirical data, where the empirical data points are the extrapolated trajectories of the current state. Considering the different focuses of the two methods, it is necessary to extrapolate the corresponding empirical data separately. Specifically, the trajectory points generated by MPI extrapolation are... The trajectory points generated by SFA extrapolation are: Calculate the extrapolation control strategies for both methods at the extrapolated trajectory points. and extrapolation Bellman error δ 1,k (t),δ 2,j(t). Adaptive update rules for the evaluation and behavior networks are determined using gradient descent and gradient projection operators, i.e., the method for weight update iteration. Then, composite obstacle avoidance is achieved through reinforcement learning, where the evaluation and behavior networks respectively perform policy evaluation and policy improvement. Under the action of a switching function, the two pairs of ACNN structures are switched appropriately to generate continuous cost estimates and control outputs.
[0117] Example 2
[0118] The following example, using specific calculation formulas, further illustrates the scheme in example 1 in example 2, as detailed below:
[0119] First, through Figure 1 Step S10 completes the construction of the obstacle avoidance environment and optimal control model.
[0120] S10: Construct the obstacle avoidance environment and the optimal control model for obstacle avoidance;
[0121] In this embodiment, it can be achieved through Figure 2 The steps in the process of constructing the obstacle environment and optimal control model include: Step S10 includes:
[0122] Step S101: Construct an obstacle environment around the obstacle. Considering the integrated control and decision-making of autonomous unmanned systems and their inherent complex nonlinearity, the autonomous unmanned system is regarded as an intelligent agent and represented by the following affine nonlinear model:
[0123]
[0124] Where f(x(t)) represents internal dynamic information, g(x(t)) represents the control coupling matrix, and u(t) represents the control input or control strategy. This agent possesses local detection capability, with a detection radius of D. i The intelligent agent can use obstacle detectors to detect the real-time distance to each obstacle. in This represents the position of the i-th obstacle; the goal of obstacle avoidance control is to allow the agent to safely bypass all obstacles and reach the target point x. e Treating obstacles as circular or spherical regions, constructing an obstacle environment around the obstacle, and establishing the corresponding obstacle region O. i (radius is r) i Based on this, conflict zone C radiates outwards. i (radius is R) i Avoidance zone (radius is L) i ) and switching areas (radius is D) i The number of obstacles in the obstacle avoidance environment is denoted as N.o Then the area faced by the intelligent agent in obstacle avoidance can be represented as Specifically, after entering the switching zone, the agent will encounter obstacles, at which point the MPI method will gradually switch to the SFA method; after entering the avoidance zone, it will fully switch to the SFA method, and the agent will begin to take obstacle avoidance actions; after entering the conflict zone, it will take larger avoidance actions, and to ensure safety, the agent cannot go too deep into this zone; the obstacle zone is the danger zone, and once the agent enters it, it means that the mission has failed.
[0125] Step S102: Define the safe operating domain and design the obstacle function. Based on the above, the obstacle region can be represented using forward invariance as follows: Where h(x) is a smooth function, the safe operating region of the system is S = Ψ - 0, where Ψ represents the operating space of the system. Based on the safe operating region, using the barrier function method, the following barrier function B(x) is designed:
[0126]
[0127] Where K b It is a positive gain, and s(x) is a smoothly varying scheduling function whose value smoothly changes between 0 and 1. Clearly, the parameter K... b The magnitude of the barrier function is adjusted, and s(x) determines whether the barrier function can function.
[0128] Step S103: Establish an optimal control model based on the cost function. In some embodiments, the system operating cost x is comprehensively considered. T Qx, controlling energy costs u T Given Ru and the barrier function B(x), the following cost function to be optimized is established:
[0129]
[0130] Where Q and R are positive definite and symmetric constant matrices used to measure the magnitude of specific cost terms. Then, using the Bellman optimality principle, the optimal control model for obstacle avoidance is established as follows:
[0131]
[0132]
[0133] Among them, ▽J * It is the optimal cost function J * (x) is the partial derivative of x with respect to state x. In this optimal control model, equation (4) represents the optimal obstacle avoidance control strategy, and equation (5) represents the Bellman equation, the solution of which corresponds to the optimal cost function J. *(x). Obviously, how to calculate (4) and solve equation (5) is the key to the obstacle avoidance problem.
[0134] After obtaining the optimal control model, it is possible to continue through... Figure 1 Step S20 in the design process involves a composite obstacle avoidance control method that allows for a reasonable switch between the MPI and SFA methods.
[0135] S20: Design method switching rules to achieve composite obstacle avoidance control;
[0136] In this embodiment, it can be achieved through Figure 3 The steps in the process achieve composite obstacle avoidance control, such as Figure 3 As shown, step S20 includes:
[0137] Step S201: Construct the overall framework of the composite obstacle avoidance control method. Considering the obstacle environment and the optimal control model, a linear convex combination of the MPI and SFA methods is used to construct the overall framework of the composite obstacle avoidance control method. Mathematically, the cost function of the agent is calculated as follows:
[0138]
[0139] Where λ(x) is the scheduling function, It is the estimate of the cost function by the MPI method, while This is the cost function estimate obtained by the SFA method. In this composite framework, the continuity of the cost function during method switching can be guaranteed by using the linearized convex combination shown in Equation (6).
[0140] Step S202: Design the switching rules for the MPI method. The switching rules for the MPI method mainly consist of the switching function λ(x), which is implemented using a cosine function and calculated as follows:
[0141]
[0142] Where λ(x) contains N o There are 1 component, each smoothly varying between 0 and 1. If obstacle i is never within the detection range of the agent, then the corresponding component λ... i (x) will never be activated, i.e., λ i (x) = 0.
[0143] Step S203: Design the switching rules for the SFA method. The switching rules for the SFA method mainly consist of the scheduling function s(x), which is similarly implemented using a cosine function and calculated as follows:
[0144]
[0145] Wherein, s(x) also contains N o There are several components, each smoothly varying between 0 and 1. Clearly, if the component corresponding to obstacle i is λ... i If (x) is not activated, then s i (x) is also never activated; as the agent approaches the obstacle region, the scheduling function will approach 1, which means that the obstacle function has a greater impact on the cost function. This will conflict with the policy optimization behavior, and thus cause the agent to take avoidance behavior in subsequent actions.
[0146] In the constructed composite obstacle avoidance control method, both the MPI and SFA methods require estimates of the cost function. To achieve this, the following steps are performed: Figure 1 Step S30 in the process.
[0147] S30: Construct an obstacle avoidance control strategy based on behavior-evaluation neural networks and reinforcement learning;
[0148] In this embodiment, it can be achieved through Figure 4 The steps in the process involve constructing a neural network and obtaining an obstacle avoidance control strategy, such as... Figure 4 As shown, step S30 mainly includes:
[0149] Step S301: Construct the behavior-evaluation neural network structure. Design the obstacle penalty term B based on the obstacle function B(x). o (x) is used to characterize the reward and punishment situation of the intelligent agent in obstacle avoidance, and is designed as follows:
[0150]
[0151] Where α is a positive constant. It's easy to see from this obstacle avoidance penalty term that it becomes infinitely large when the agent approaches the obstacle area, and gradually approaches zero when moving away, thus achieving the "reward and punishment" for obstacle avoidance actions. Based on this, an evaluation network and a behavior network are constructed using a multi-layer feedforward neural network. It's important to note that ACNN structures need to be designed separately for MPI and SFA, resulting in an agent containing two evaluation networks and two behavior networks. For the two evaluation networks, the evaluation network for the MPI method is constructed as follows: Among them is The evaluation weights are σ(x), which is the corresponding activation function; the evaluation network of the SFA method is constructed as follows: in It is an evaluation weight. c(x) is the state-following basis function, and c(x) is the corresponding kernel function. Based on this, using the linear convex combination of formula (6), the approximate value of the cost function is obtained as follows:
[0152]
[0153] In the above formula, λ(x) is simplified to λ. For behavioral networks, a similar structure is used, with its weights... Used to estimate the ideal evaluation weight w c1 ,w c2 This enables interaction between the two networks.
[0154] Step S302: Obtain the obstacle avoidance control strategy based on reinforcement learning. Based on the constructed ACNN structure, obtain the obstacle avoidance control strategy based on reinforcement learning. The calculation is as follows:
[0155]
[0156] It can be seen that the obstacle avoidance control strategy includes the weight parameters of two behavioral networks, and this control strategy is also the actual output signal of the controller.
[0157] Step S303: Calculate the real-time Bellman error. The obtained cost function... and obstacle avoidance control strategies Substituting into the Bellman equation (5), we obtain the real-time Bellman error δ(t) as follows:
[0158]
[0159] Bellman error encompasses all neural network parameters and characterizes the learning error in reinforcement learning.
[0160] Having obtained real-time learning data, to ensure obstacle avoidance effectiveness, it is still necessary to use extrapolation to obtain experience data for autonomous exploration. Therefore, we will continue to use... Figure 1 Step S40 in the process implements experience playback and determines network update rules.
[0161] S40: Use extrapolation to obtain empirical data and determine adaptive network update rules;
[0162] In this embodiment, it can be achieved through Figure 5 The steps in the process include acquiring empirical data and updating rule design, such as... Figure 5 As shown, step S40 mainly includes:
[0163] Step S401: Obtain empirical data using the state extrapolation method. To avoid potential exploration hazards, empirical data is obtained using the state extrapolation method. The empirical data points are the extrapolated trajectories of the current state. Considering the different focuses of the two methods, it is necessary to extrapolate and obtain the corresponding empirical data separately. Specifically, the trajectory points generated by MPI extrapolation are... The trajectory points generated by SFA extrapolation are: Based on the form of (4) in the optimal control model, calculate the extrapolation control strategy.
[0164]
[0165] Based on the real-time Bellman error in formula (12), the extrapolated Bellman error δ is calculated. 1k (t),δ 2j (t):
[0166]
[0167] Step S402: Determine the adaptive update rules for the evaluation and behavior networks. Combining real-time data and empirical data generated through extrapolation, establish the following sum of learning errors:
[0168]
[0169] To minimize the learning error in the above equation, the adaptive update rule for the evaluation network is determined using the gradient descent adaptive method as follows:
[0170]
[0171] Where k c11 ,k c12 >0 is the evaluation network learning rate for the MPI method, k c21 ,k c22 >0 represents the evaluation network learning rate for the SFA method. The network parameter updates for the two methods are independent, each using its own extrapolated empirical data, but both require real-time data. In some embodiments, to prevent control divergence caused by excessively large behavioral weights, the gradient projection operator method is used to constrain the weights, resulting in the following adaptive update rule for the behavioral network:
[0172]
[0173] Where proj{·} is the projection operator, and k is the projection operator. a1 >0 represents the learning rate of the behavior network in the MPI method, k a2 >0 is the learning rate of the behavioral network in the SFA method, F a1 ,F a2 It is a positive definite constant matrix.
[0174] Step S403: Composite obstacle avoidance is achieved through reinforcement learning. After completing the above steps, the policy iteration process of the reinforcement learning algorithm is executed by the evaluation network and the behavior network, where the evaluation network and the behavior network respectively perform policy evaluation and policy improvement. These two processes are further manifested as adaptive update processes (16) and (17). Under the action of the switching function, the two pairs of ACNN structures are switched reasonably to generate continuous cost estimates, giving the cost function of the composite obstacle avoidance method; under the action of the scheduling function, the obstacle penalty term ensures the safety of the control policy, giving the control policy of the composite obstacle avoidance method, and realizing autonomous obstacle avoidance.
[0175] Example 3
[0176] The feasibility of the schemes in Examples 1 and 2 is verified below using specific experimental data and examples. These examples are conducted on an autonomous vehicle system, as detailed in the following description:
[0177] According to step S10, five obstacles were set up, and the detection radius of the unmanned vehicle is 1 meter, i.e., D. i = 1m, the radii of the corresponding areas radiated by each obstacle are r1 = 0.15, R1 = 0.2, L1 = 0.95; r2 = 0.1, R2 = 0.15, L2 = 0.95; r3 = 0.12, R3 = 0.17, L3 = 0.95; r4 = 0.12, R4 = 0.17, L4 = 0.95; r5 = 0.15, R5 = 0.2, L5 = 0.95; the coordinates of the five obstacles are... The gain of the barrier function B(x) is K b =0.95; the cost function is configured as Q=I², R=5I²; finally, the starting point of the autonomous vehicle is set to (1.2, 1.5), and the target point x e It is calibrated as (0,0).
[0178] According to step S20, the switching rules for MPI and SFA methods are calculated according to formulas (7) and (8).
[0179] According to step S30, in this embodiment, three nodes are configured in the hidden layers of the behavior network and the evaluation network. Specifically, the activation function of the MPI method is set to... The state-following basis function of the SFA method is set to Without loss of generality, the three kernel functions can be chosen as follows:
[0180]
[0181] In addition, obstacle avoidance penalty item B o The parameter in (x) is set to α = 1.
[0182] According to step S40, the virtual extrapolation trajectories of both methods randomly generate 25 data points in a uniform distribution, i.e., M=25, N=25. The parameter selection in the ACNN structure update rule is k. c11 =0.1,k c12 =1.5,k a1 =1.2,F a1 =0.1I3 k c21 =0.1,k c22 =0.75,k a2 =0.75,F a2 =0.1I3.
[0183] Figure 6 This is a safe trajectory diagram illustrating the obstacle avoidance process of the autonomous vehicle system according to this embodiment. It can be seen that the autonomous vehicle can effectively avoid five obstacles and successfully reach the target point, generating a smooth and safe trajectory. The trajectory of the autonomous vehicle in the switching area is also relatively smooth, demonstrating the effectiveness of the composite obstacle avoidance method. Furthermore, the autonomous vehicle basically does not enter the conflict area, indicating that the design of the obstacle penalty item is reasonable and effective. Figure 7 This example illustrates the changes in the switching function λ(x) and the scheduling function s(x). It can be seen that the switching function changes smoothly between 0 and 1, indicating that both MPI and SFA methods operate for a period of time, with the SFA method only operating during obstacle avoidance. The scheduling function changes only within a very small range, demonstrating that the obstacle penalty is effective in penalizing and rewarding the autonomous vehicle, ensuring that the vehicle never approaches obstacles and guaranteeing its safety.
[0184] Example 4
[0185] Based on Example 3, other obstacle avoidance control methods are considered below. The reliability and advantages of the reinforcement learning-based composite obstacle avoidance control method disclosed herein are verified through comparison. See the description below for details:
[0186] In this embodiment, the composite obstacle avoidance method is implemented according to the steps in Embodiment 3. In addition, three other obstacle avoidance control schemes are considered, referred to as Comparison Method 1, Comparison Method 2, and Comparison Method 3.
[0187] Figure 8 This is a comparison diagram of the safe trajectories of the obstacle avoidance process of the unmanned vehicle system according to this embodiment. It can be seen that the motion trajectory generated by the composite obstacle avoidance control method of this disclosure is smoother and has less oscillation, indicating better optimization of the strategy; the unmanned vehicle does not go too deep into the conflict area, indicating better obstacle avoidance safety; the unmanned vehicle has the shortest motion trajectory, indicating that the MPI and SFA methods have a good balance between safety and optimization.
[0188] Based on the results of Examples 3 and 4, the beneficial effects of the composite obstacle avoidance control method based on reinforcement learning disclosed in this invention are indeed real.
[0189] Example 5
[0190] This embodiment illustrates a composite obstacle avoidance control device based on reinforcement learning. See [link to relevant documentation]. Figure 9 The device D90 in the middle includes the following components:
[0191] The obstacle detector D901 is used to detect the distance between unmanned systems and obstacles. For example, the obstacle detector can be a lidar, but its detection range is limited, with a detection radius of D. i The D901 is responsible for detecting the real-time distance d between the unmanned vehicle and obstacles. i This information is then fed back to the microprocessor to calculate the switching function λ(x) and the scheduling function s(x).
[0192] The D902 microprocessor, primarily consisting of embedded chips and related functional circuits, is used to perform neural network training and control calculations. Both the evaluation network and the behavioral network are implemented within it in code form. The output signal of the D902 constitutes the obstacle avoidance control strategy. It is calculated from two behavioral networks.
[0193] The D903 memory, interconnected with the microprocessor, can store the motion trajectory data x(t) of the unmanned system and the learning data generated by the microprocessor, mainly the weight parameters of the neural network. It is also responsible for periodically transmitting stored experience data to the D902 microprocessor.
[0194] Drive components D904 and D905 are responsible for executing control commands output by the microprocessor, thereby providing power to the unmanned system. For example, in embodiments 3 and 4, the drive components are micro-motor devices with external wheels; in some embodiments, the drive components may also have external rotors to form an unmanned aerial vehicle (UAV) system. Note that, without limitation, there may be more drive components, but for the method of this disclosure, the autonomous unmanned system needs to have symmetrical drive components.
[0195] It should be noted that the device descriptions in the above embodiments correspond to the method descriptions in the embodiments, and the embodiments of the present invention will not be repeated here.
[0196] Those skilled in the art will understand that the embodiments of this disclosure can be provided as methods, prototype systems, autonomous intelligent mobile platforms, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or a combination of hardware and software embodiments.
[0197] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A composite obstacle avoidance control method based on reinforcement learning, characterized in that, The method includes the following steps: S10. Construct an obstacle avoidance environment and an optimal control model for obstacle avoidance; S20. Based on the obstacle environment and the optimal control model, a composite obstacle avoidance control model is constructed by switching between a linear convex combination of the strategy iteration method and the state following avoidance method; the composite obstacle avoidance control model is as follows: Where λ(x) is the switching function, and its value also changes smoothly between 0 and 1; It is the estimate of the cost function by the strategy iteration method; It is the estimate of the cost function by the state-following avoidance method; S30. Construct an obstacle avoidance control strategy based on behavior-evaluation neural networks and reinforcement learning; S40. Use extrapolation to obtain empirical data and determine adaptive network update rules.
2. The composite obstacle avoidance control method based on reinforcement learning according to claim 1, characterized in that, The process of constructing the obstacle avoidance environment and the optimal control model for obstacle avoidance in step S10: Step S101: Construct an obstacle environment around the obstacle object; that is: The intelligent agent of the autonomous unmanned system is characterized by the following affine nonlinear model: Where: f(x(t)) is the internal dynamic information, g(x(t)) represents the control coupling matrix, and u(t) represents the control input or control strategy; The intelligent agent uses an obstacle detector to detect the real-time distance to each obstacle. Its control objective is to safely bypass every obstacle and reach the target point x. e The number of obstacles is denoted as N. o ; The obstacle avoidance environment treats obstacles as circular or spherical areas, establishing corresponding obstacle areas O. i Based on this, conflict zones radiate outwards. Avoidance area and switch regions The area faced by an intelligent agent in obstacle avoidance can be represented as Step S102: Construct a safe operating domain and design barrier functions; Using forward invariance, the obstacle region can be represented as Where h(x) is a smooth function, and the safe operating domain of the system is S = Ψ - 0, where Ψ represents the operating space of the system; Based on the safe operating domain, the following barrier function B(x) is designed using the barrier function method: Where: K b It is a positive gain, and s(x) is a smoothly changing scheduling function whose value changes smoothly between 0 and 1; Step S103: Establish an optimal control model based on the cost function; System operating cost x T Qx, controlling energy costs u T Given Ru and the barrier function B(x), the following cost function to be optimized is established: Where Q and R are positive definite and symmetric constant matrices used to measure the size of specific cost items; Using the Bellman optimality principle, the optimal control model for obstacle avoidance is established as follows: in: It is the optimal cost function J * (x) is the partial derivative of (x) with respect to the state x.
3. The composite obstacle avoidance control method based on reinforcement learning according to claim 1, characterized in that, The rules for the linear convex combination of the policy iteration method and the state following avoidance method in step 20 are as follows: The strategy iteration method implements the switching function through the following formula, which affects when the strategy iteration method starts to switch to the state following avoidance method. Where: λ(x) contains N o There are 1 component, each smoothly varying between 0 and 1. If obstacle i is never within the detection range of the agent, then the corresponding component λ... i (x) will never be activated, i.e., λ i (x) = 0; The state-following avoidance method implements the scheduling function through the following formula, which affects the obstacle avoidance process of the optimal control model; Wherein, s(x) also contains N o There are 1 component, each of which smoothly varies between 0 and 1; if the component corresponding to obstacle i is λ i If (x) is not activated, then s i (x) is also never activated; as the agent approaches the obstacle region, the scheduling function will approach 1, which means that the obstacle function has a greater impact on the cost function. This will conflict with the policy optimization behavior, and thus cause the agent to take avoidance behavior in subsequent actions.
4. The composite obstacle avoidance control method based on reinforcement learning according to claim 1, characterized in that, The process of constructing the behavior-evaluation neural network and the obstacle avoidance control strategy through reinforcement learning in step S30: Step S301: Design an obstacle penalty term B based on the obstacle function B(x). o (x) is used to characterize the reward and punishment situation of the intelligent agent in obstacle avoidance; on this basis, an evaluation network and a behavior network are constructed using a multi-layer feedforward neural network. Evaluation-behavior network structures are designed for the policy iteration method and the state following avoidance method, respectively. The agent contains two evaluation networks and two behavior networks. Right now: The evaluation network for the policy iteration method is constructed as follows Among them: is The evaluation weights are σ(x), which is the corresponding activation function. The evaluation network construction of the state-following avoidance method is as follows in: It is an evaluation weight. Here, c(x) is the state-following basis function, and c(x) is the corresponding kernel function. Based on this, through a linear convex combination of the following formulas, the approximate value of the cost function is obtained as follows: In the above formula, λ(x) is simplified to λ; for behavioral networks, a similar structure is adopted, with its weights... Used to estimate the evaluation weight w c1 ,w c2 That is: obstacle avoidance control strategy based on reinforcement learning Among them: the obstacle avoidance control strategy includes the weight parameters of two behavioral networks; Step S302: Apply the Bellman equation to the cost function. and obstacle avoidance control strategies The real-time Bellman error δ(t) is calculated as follows: Among them, Bellman error includes all neural network parameters and represents the learning error in reinforcement learning.
5. The composite obstacle avoidance control method based on reinforcement learning according to claim 1, characterized in that, The process of using extrapolation to obtain empirical data and determine adaptive network update rules in step S40 is as follows: Step S401: Obtain empirical data using the state extrapolation method; The trajectory points generated by extrapolation using the strategy iteration method are: The trajectory points generated by the state-following avoidance method extrapolation are: Computational extrapolation control strategy The trajectory points generated by extrapolation using the strategy iteration method are: The trajectory points generated by the state-following avoidance method extrapolation are: Calculate the extrapolation Bellman error δ 1,k (t),δ 2,j (t): Step S402: Determine the adaptive update rules for the evaluation and behavior networks using the gradient descent adaptive method and the gradient projection operator method; that is: The adaptive update rule for the evaluation network obtained using the gradient descent adaptive method is as follows: Where: k c11 ,k c12 >0 is the evaluation network learning rate for the policy iteration method, k c21 ,k c22 >0 is the evaluation network learning rate for the state-following avoidance method; The adaptive update rule for the behavioral network obtained using the gradient projection operator method is as follows: Where: proj{·} is the projection operator, k a1 >0 represents the learning rate of the behavioral network in the policy iteration method, k a2 >0 represents the learning rate of the behavioral network using the state-following avoidance method, F. a1 ,F a2 It is a positive definite constant matrix; Step S403: Composite obstacle avoidance is achieved through reinforcement learning; wherein the evaluation network and the behavior network perform policy evaluation and policy improvement respectively, and under the action of the switching function, the two pairs of evaluation-behavior network structures are switched reasonably to generate continuous cost estimates and control outputs.
6. A composite obstacle avoidance control device based on reinforcement learning, characterized in that, The device includes: An obstacle detector is used to detect the distance between an unmanned system and obstacles and feeds this information back to the microprocessor; however, this detector has a limited detection range, i.e., a finite detection radius. The microprocessor, mainly consisting of embedded chips and related functional circuits, is used to execute any one of the steps of claims 1-5, and its output signal is the obstacle avoidance control strategy obtained by behavior network calculation. The memory, interconnected with the microprocessor, can save the motion trajectory data of the unmanned system, save the learning data generated by the microprocessor, and also periodically transmit the stored experience data to the microprocessor. The drive component is responsible for executing the control commands output by the microprocessor, thereby providing power to the unmanned system.