Charging pile positioning method based on hierarchical reinforcement learning

Abstract

The invention relates to the technical field of indoor unmanned vehicle autonomous charging, in particular to a layered reinforcement learning-based charging pile positioning method, which comprises the steps of constructing a layered reinforcement learning framework based on a stage management module and a dynamic control module; the stage management module is used for dividing a plurality of sub-tasks by using a high-level policy network and performing state evaluation on the sub-tasks; the dynamic control module is used for executing specific actions of the subtasks by utilizing a bottom-layer strategy network; and executing a plurality of subtasks based on the hierarchical reinforcement learning framework, including: controlling the robot to go to the front of the charging pile, rotating the robot to face the charging pile, performing line tracking based on the reflector data, and rotating the robot to make the electrode aligned with the charging pile and retreat to a final target point. According to the invention, the positioning precision, the stability and the adaptability to a complex indoor environment are improved.

Description

Technical Field

[0001] This invention relates to the field of indoor unmanned vehicle autonomous charging technology, specifically a charging pile positioning method based on hierarchical reinforcement learning. Background Technology

[0002] Mobile robots, as the earliest type of robot to achieve commercial application, have been developed over decades and are now widely used in various fields such as industrial manufacturing, logistics and warehousing, medical services, and home services. With the exponential growth in the number of deployed mobile robots and the continuous improvement in their intelligence level, autonomous charging has become an important function, especially in unattended operation scenarios. Currently, the main methods for locating charging stations for mobile robots include infrared sensor guidance, pure vision guidance, infrared plus vision, and LiDAR plus vision. The basic principle is to locate the robot by recognizing reflectors, QR codes, or some other special structure on the charging station.

[0003] Traditional LiDAR and camera-based charging station positioning methods suffer from low computational efficiency and insufficient real-time performance; LiDAR's threshold for recognizing reflectors cannot be dynamically changed, resulting in poor environmental robustness. Summary of the Invention

[0004] The purpose of this invention is to propose a charging pile positioning method based on hierarchical reinforcement learning. In order to solve the above problems, this invention adopts a hierarchical reinforcement learning framework, which decomposes the decision-making process into a stage management module and a dynamic control module to achieve intelligent optimization of the positioning process. By adjusting the dynamic parameters, the system's adaptability to different environments is enhanced.

[0005] To solve the above-mentioned technical problems, the present invention provides the following technical solution:

[0006] A charging station localization method based on hierarchical reinforcement learning, the method includes:

[0007] S1. A hierarchical reinforcement learning framework is constructed based on a stage management module and a dynamic control module; the stage management module is used to divide multiple sub-tasks using a high-level policy network and to evaluate the state of the sub-tasks; the dynamic control module is used to execute the specific actions of the sub-tasks using a low-level policy network.

[0008] S2. Based on a hierarchical reinforcement learning framework, perform multiple sub-tasks, including: controlling the robot to move directly in front of the charging pile, rotating the robot to face the charging pile, performing line tracking based on reflector data, and rotating the robot to align the electrodes with the charging pile and then moving back to the final target point.

[0009] Preferably, the phase management module in S1 is used to divide multiple subtasks using a high-level policy network and to perform state evaluation of the subtasks, including:

[0010] The high-level policy network treats subtasks in the localization process as options. The option Represented as a triple ;in, express Initial state set, Indicates the termination condition. Indicates the current Strategies;

[0011] Using a semi-Markov decision model Layered reinforcement, then given a strategy Under, state The value function is: ;

[0012] in, Representing state The value function, Indicates the state Execute action Real-time rewards Indicates the discount factor. Indicates the duration of the action. Indicates the state after the transition;

[0013] Further in the state Choose one The formula for the total revenue generated at that time is as follows: ;

[0014] in, These represent different parameter sets. Indicates state, It represents a specific option.

[0015] Preferably, the dynamic control module in S1 is used to execute specific actions of the subtask using the underlying policy network, including:

[0016] The underlying strategy is responsible for each The robot then performs specific actions, the selection and execution of which are based on the reflector data identified by the lidar, the robot's current pose, and the sub-task objective.

[0017] The policy network is based on the current local state Select an action and confirm the action. And based on the selected action The system drives the autonomous vehicle and simultaneously calculates state deviations in real time using radar data. ;

[0018] If the deviation exceeds the tolerance range This triggers the policy network to adjust the action distribution online until the desired outcome is achieved. Termination conditions ; in choosing a Under the premise of taking a certain The resulting total revenue function is as follows: ;

[0019] in, Indicates the state Choose the option below Execute action at time value, Indicates the discount factor. Let be the state transition probability, representing the probability in state . Next action When the state is reached The probability of; Indicates based on the current The expected rewards afterward.

[0020] Preferably, the first subtask in S2, controlling the robot to move directly in front of the charging station, includes:

[0021] The high-level strategy obtains local target parameters, including the coordinates of the area directly in front of the charging pile. Allowable distance error and angle error Confirm when distance error and angle error The current option is considered complete only when the tolerance range is not exceeded.

[0022] Read the robot's current state using the underlying strategy Combine LiDAR data and initialize its own cost function parameters; define the current... During execution, state The immediate cost is:

[0023] ;

[0024] in, This represents the distance error, calculated from the coordinates of the temporary target point in the robot's base coordinate system. The angular error is also calculated from the coordinates of the temporary target point in the robot's base coordinate system. , , This represents the weighting factor, initialized by the underlying policy, where t represents the current time.

[0025] The action sequence is then found by minimizing the cumulative cost:

[0026] ;

[0027] in, Indicates the discount factor, the further down the line... The smaller;

[0028] Through the trained policy network PPO Output the optimal action: ;

[0029] When the termination condition is met, the lower-level option sends a "completion signal" to the higher-level policy and provides feedback. The cumulative cost, followed by the high-level strategy continuing to choose the next... .

[0030] Preferably, the second subtask in S2, rotating the robot to face the charging station, includes:

[0031] The subtask The rotating robot faces the charging station. Considering only the angle error, the cumulative cost of the entire rotation process is defined as the cumulative cost of the rotation angle error:

[0032] ;

[0033] in, This indicates the angle deviation at the current moment, determined based on the position of the final target point in the robot's base coordinate system. Indicates the weighting coefficient. Indicates the angle at the current moment;

[0034] Further minimize the cost at the current time step using the policy network. To output angular velocity ;in, Indicates the proportionality coefficient;

[0035] When the deviation is within the tolerance range, a signal is sent to the higher-level policy to complete the current option.

[0036] Preferably, the third subtask in S2, line tracking based on reflector data, includes:

[0037] After the robot rotates to the correct orientation, linear tracking is performed, and the desired rotation angle is calculated according to the Stanley algorithm. : ;

[0038] in, This represents the heading error, which is the difference between the robot's current heading angle and the desired heading angle. Indicates lateral error. This represents the proportionality coefficient, used to adjust the strength of the correction for lateral errors. Indicates the robot's moving speed. Represents a constant;

[0039] Define the current Cost function: ;

[0040] in, , , Indicates the weighting coefficient. This indicates the robot's maximum speed during movement;

[0041] Furthermore, the cost function is minimized using gradient descent through the built-in optimizer, the linear velocity is iteratively updated, and the desired steering angle is converted into angular velocity based on the robot's wheelbase L. The specific calculation formula is as follows:

[0042] ;

[0043] ;

[0044] in, This represents the learning rate, which controls the step size of gradient descent. This represents the partial derivative of the cost function with respect to the linear velocity at the current moment. This represents the damping coefficient, used to reduce high-frequency jitter in angular velocity.

[0045] Preferably, the fourth subtask in S2, rotating the robot to align the electrodes with the charging station and retract to the final target point, includes:

[0046] As the final execution link of the underlying policy network, this It will perform two consecutive actions: rotation and backward movement. Closed-loop control is achieved through a hierarchical reinforcement learning policy network. The cost functions for the two independent actions and the final total cost function are as follows:

[0047] ;

[0048] ;

[0049] ;

[0050] in, This represents the action weighting coefficient, used to distinguish the priority of rotation and backward movement. At that time, only the rotation cost is optimized. When the value is 0, only the backoff cost is optimized; , , , Indicates the proportionality coefficient;

[0051] The angle and distance deviations calculated based on LiDAR or visual data are used as inputs to the underlying policy network. The network output layer then minimizes the total cost function. Generate control variables;

[0052] The underlying network first predicts a temporary angular velocity. Calculate its rotation cost The gradient of the cost with respect to the predicted value is calculated using backpropagation. The final output is the angular velocity corrected by gradient descent:

[0053] ;

[0054] in, The learning rate is used to ensure Adjust in the direction of least cost;

[0055] Once the rotation is complete, the motion weighting coefficient... When the value is changed to 1, the network predicts a temporary linear velocity. Calculate the backoff cost gradient of cost with respect to predicted value Output the corrected linear velocity:

[0056] ;

[0057] When the underlying policy network detects that the last option meets the termination condition, it determines that the entire process of positioning is complete.

[0058] At this point, the upper-layer policy network records the total cost and task completion status of this localization, triggering the global update mechanism of reinforcement learning: adjusting the weights of the upper-layer policy based on the deviation between the total cost and the preset optimal threshold, while feeding back the local optimization results of each option to the parameter update process of the lower-layer policy network, thereby optimizing the entire localization system.

[0059] Compared with the prior art, the beneficial effects achieved by the present invention are:

[0060] This invention first employs a high-level strategy network for task scheduling and status evaluation, dynamically dividing sub-task modules, setting module activation conditions, and evaluating the execution status of sub-tasks in real time. The lower-level strategy network handles data parsing and sub-task execution, making the overall positioning process more rational and improving positioning accuracy, stability, and adaptability to complex indoor environments. Furthermore, the strategy network can continuously learn and optimize decision-making and data processing methods; with increasing positioning frequency, the positioning effect gradually improves, demonstrating self-learning and self-optimization capabilities. Attached Figure Description

[0061] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0062] Figure 1 This is a flowchart of a charging pile positioning method based on hierarchical reinforcement learning according to the present invention;

[0063] Figure 2 This is a diagram of the hierarchical reinforcement learning framework in an embodiment of the present invention. Detailed Implementation

[0064] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0065] Please see Figures 1-2 The present invention provides the following technical solution:

[0066] Example 1: As Figure 1 As shown, a charging station localization method based on hierarchical reinforcement learning is proposed, the method comprising:

[0067] Hierarchical reinforcement learning can decompose complex problems into multiple smaller problems, and solve the original problem by solving each smaller problem separately. This avoids the problems of traditional reinforcement learning, such as the agent's state space being too large and the storage space being too large due to the complexity of the problem.

[0068] S1, such as Figure 2 As shown, a hierarchical reinforcement learning framework is constructed based on the stage management module and the dynamic control module; the hierarchical approach is adopted to divide the robot's target location process into the stage selection of high-level strategies and the specific control of low-level strategies.

[0069] The phase management module is used to divide multiple sub-tasks using a high-level policy network and to evaluate the status of the sub-tasks; the dynamic control module is used to execute the specific actions of the sub-tasks using a low-level policy network.

[0070] Preferably, the phase management module in S1 is used to divide multiple subtasks using a high-level policy network and to perform state evaluation of the subtasks, including:

[0071] The high-level policy network treats subtasks in the localization process as options. The option It can be viewed as an abstraction of an action, represented by a triple. ;in, express Initial state set, Indicates the termination condition. Indicates the current Strategies;

[0072] Using a semi-Markov decision model Hierarchical reinforcement, this model allows options to have variable durations, similar to the hierarchical reinforcement learning process. The time inconsistency is adapted; the value function of SMDP can be extended to a semi-Markov form by the Bellman equation, then given a strategy Under, state The value function is: ;

[0073] in, Representing state The value function, Indicates the state Execute action Real-time rewards Indicates the discount factor. Indicates the duration of the action. Indicates the state after the transition;

[0074] Further in the state Choose one The formula for the total revenue generated at that time is as follows: ;

[0075] in, These represent different parameter sets. Indicates state, It represents a specific option.

[0076] Preferably, the dynamic control module in S1 is used to execute specific actions of the subtask using the underlying policy network, including:

[0077] The underlying strategy is responsible for each The robot then performs specific actions, the selection and execution of which are based on the reflector data identified by the lidar, the robot's current pose, and the sub-task objective.

[0078] The policy network is based on the current local state Select an action and confirm the action. And based on the selected action The system drives the autonomous vehicle and simultaneously calculates state deviations in real time using radar data. ;

[0079] If the deviation exceeds the tolerance range This triggers the policy network to adjust the action distribution online until the desired outcome is achieved. Termination conditions ; in choosing a Under the premise of taking a certain The resulting total revenue function is as follows: ;

[0080] in, Indicates the state Choose the option below Execute action at time value, γ represents the discount factor, used to balance the weight of current and future rewards. 0 < γ ≤ 1, and the closer it is to 1, the more emphasis is placed on future cumulative rewards. Let be the state transition probability, representing the probability in state . Next action When the state is reached The probability of; Indicates based on the current The expected rewards afterward.

[0081] S2. Based on a hierarchical reinforcement learning framework, perform multiple sub-tasks, including: controlling the robot to move directly in front of the charging pile, rotating the robot to face the charging pile, performing line tracking based on reflector data, and rotating the robot to align the electrodes with the charging pile and then moving back to the final target point.

[0082] Preferably, the first subtask in S2, controlling the robot to move directly in front of the charging station, includes:

[0083] The high-level strategy obtains local target parameters, including the coordinates of the area directly in front of the charging pile. Allowable distance error and angle error Confirm when distance error and angle error The current option is considered complete only when the tolerance range is not exceeded.

[0084] Read the robot's current state using the underlying strategy Combine LiDAR data and initialize its own cost function parameters; define the current... During execution, state The immediate cost is:

[0085] ;

[0086] in, This represents the distance error, calculated from the coordinates of the temporary target point in the robot's base coordinate system. The angular error is also calculated from the coordinates of the temporary target point in the robot's base coordinate system. , , This represents the weighting factor, initialized by the underlying policy, where t represents the current time.

[0087] The action sequence is then found by minimizing the cumulative cost:

[0088] ;

[0089] in, Indicates the discount factor, the further down the line... The smaller the value, the more the agent focuses on short-term costs, which is beneficial for accumulation and convergence;

[0090] Through the trained policy network PPO Output the optimal action: ;

[0091] When the termination condition is met, the lower-level option sends a "completion signal" to the higher-level policy and provides feedback. The cumulative cost, followed by the high-level strategy continuing to choose the next... .

[0092] Preferably, the second subtask in S2, rotating the robot to face the charging station, includes:

[0093] The subtask The rotating robot faces the charging station. Considering only the angle error, the cumulative cost of the entire rotation process is defined as the cumulative cost of the rotation angle error:

[0094] ;

[0095] in, This indicates the angle deviation at the current moment, determined based on the position of the final target point in the robot's base coordinate system. This represents the weighting coefficient, used to suppress large-amplitude rotations and ensure the smoothness of the rotation process. Indicates the angle at the current moment;

[0096] Further minimize the cost at the current time step using the policy network. To output angular velocity ;in, This represents a scaling factor, which makes the network more inclined to output actions that reduce costs under similar conditions;

[0097] When the deviation is within the tolerance range, a signal is sent to the higher-level policy to complete the current option.

[0098] Preferably, the third subtask in S2, line tracking based on reflector data, includes:

[0099] After the robot rotates to the correct orientation, linear tracking is performed, and the desired rotation angle is calculated according to the Stanley algorithm. The Stanley algorithm is a classic path tracking algorithm that is based on a geometric model and adjusts the vehicle's steering angle using lateral and heading errors to achieve efficient and stable path tracking.

[0100] ;

[0101] in, This represents the heading error, which is the difference between the robot's current heading angle and the desired heading angle. Indicates lateral error. This represents the proportionality coefficient, used to adjust the strength of the correction for lateral errors. Indicates the robot's moving speed. It represents a very small constant to avoid the denominator being 0 when the velocity is zero;

[0102] Define the current Cost function: ;

[0103] in, , , These represent weighting coefficients, used to penalize lateral errors, heading errors, and to encourage the robot's speed to be as close as possible to the maximum safe speed. This indicates the robot's maximum speed during movement;

[0104] Furthermore, the cost function is minimized using gradient descent through the built-in optimizer, the linear velocity is iteratively updated, and the desired steering angle is converted into angular velocity based on the robot's wheelbase L. The specific calculation formula is as follows:

[0105] ;

[0106] ;

[0107] in, The learning rate controls the step size of gradient descent and determines the magnitude by which the linear velocity is adjusted towards the optimal value in each iteration. This represents the partial derivative of the cost function with respect to the linear velocity at the current moment, reflecting the changing trend of the cost function as the linear velocity changes. This represents the damping coefficient, used to reduce high-frequency jitter in angular velocity.

[0108] Preferably, the fourth subtask in S2, rotating the robot to align the electrodes with the charging station and retract to the final target point, includes:

[0109] As the final execution link of the underlying policy network, this It will perform two consecutive actions: rotation and backward movement. Closed-loop control is achieved through a hierarchical reinforcement learning policy network. The cost functions for the two independent actions and the final total cost function are as follows:

[0110] ;

[0111] ;

[0112] ;

[0113] in, This represents the action weighting coefficient, used to distinguish the priority of rotation and backward movement. At that time, only the rotation cost is optimized. When the value is 0, only the backoff cost is optimized; , , , Indicates the proportionality coefficient;

[0114] The angle and distance deviations calculated based on LiDAR or visual data are used as inputs to the underlying policy network. The network output layer then minimizes the total cost function. Generate control variables;

[0115] The underlying network first predicts a temporary angular velocity. Calculate its rotation cost The gradient of the cost with respect to the predicted value is calculated using backpropagation. The final output is the angular velocity corrected by gradient descent:

[0116] ;

[0117] in, The learning rate is used to ensure Adjust in the direction of least cost;

[0118] Once the rotation is complete, the motion weighting coefficient... When the value is changed to 1, the network predicts a temporary linear velocity. Calculate the backoff cost gradient of cost with respect to predicted value Output the corrected linear velocity:

[0119] ;

[0120] When the underlying policy network detects that the last option meets the termination condition, it determines that the entire process of positioning is complete.

[0121] At this point, the upper-layer policy network records the total cost and task completion status of this localization, triggering the global update mechanism of reinforcement learning: adjusting the weights of the upper-layer policy based on the deviation between the total cost and the preset optimal threshold, while feeding back the local optimization results of each option to the parameter update process of the lower-layer policy network, thereby optimizing the entire localization system.

[0122] Finally, it should be noted that the above descriptions are merely preferred embodiments of the present invention and are not intended to limit the present invention. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A charging station localization method based on hierarchical reinforcement learning, characterized in that: The method includes: S1. A hierarchical reinforcement learning framework is constructed based on a stage management module and a dynamic control module; the stage management module is used to divide multiple sub-tasks using a high-level policy network and to evaluate the state of the sub-tasks; the dynamic control module is used to execute the specific actions of the sub-tasks using a low-level policy network. S2. Based on a hierarchical reinforcement learning framework, perform multiple sub-tasks, including: controlling the robot to move directly in front of the charging pile, rotating the robot to face the charging pile, performing line tracking based on reflector data, and rotating the robot to align the electrodes with the charging pile and then moving back to the final target point.

2. The charging pile positioning method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The phase management module in S1 is used to divide multiple subtasks using a high-level policy network and to evaluate the status of the subtasks, including: The high-level policy network treats subtasks in the localization process as options. The option Represented as a triple ;in, express Initial state set, Indicates the termination condition. Indicates the current Strategies; Using a semi-Markov decision model Layered reinforcement, then given a strategy Under, state The value function is: ; in, Representing state The value function, Indicates the state Execute action Real-time rewards Indicates the discount factor. Indicates the duration of the action. Indicates the state after the transition; Further in the state Choose one The formula for the total revenue generated at that time is as follows: ; in, These represent different parameter sets. Indicates state, It represents a specific option.

3. The charging pile positioning method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The dynamic control module in S1 is used to execute specific actions of subtasks using the underlying policy network, including: The underlying strategy is responsible for each The robot then performs specific actions, the selection and execution of which are based on the reflector data identified by the lidar, the robot's current pose, and the sub-task objective. The policy network is based on the current local state Select an action and confirm the action. And based on the selected action The system drives the autonomous vehicle and simultaneously calculates state deviations in real time using radar data. ; If the deviation exceeds the tolerance range This triggers the policy network to adjust the action distribution online until the desired outcome is achieved. Termination conditions ; in choosing a Under the premise of taking a certain The resulting total revenue function is as follows: ; in, Indicates the state Choose the option below Execute action at time value, Indicates the discount factor. Let be the state transition probability, representing the probability in state . Next action When the state is reached The probability of; Indicates based on the current The expected rewards afterward.

4. The charging pile positioning method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The first subtask in S2, controlling the robot to move directly in front of the charging station, includes: The high-level strategy obtains local target parameters, including the coordinates of the area directly in front of the charging pile. Allowable distance error and angle error Confirm when distance error and angle error The current option is considered complete only when the tolerance range is not exceeded. Read the robot's current state using the underlying strategy Combine LiDAR data and initialize its own cost function parameters; define the current... During execution, state The immediate cost is: ； in, This represents the distance error, calculated from the coordinates of the temporary target point in the robot's base coordinate system. The angular error is also calculated from the coordinates of the temporary target point in the robot's base coordinate system. , , This represents the weighting factor, initialized by the underlying policy, where t represents the current time. The action sequence is then found by minimizing the cumulative cost: ； in, Indicates the discount factor, the further down the line... The smaller; Through the trained policy network PPO Output the optimal action: ; When the termination condition is met, the lower-level option sends a "completion signal" to the higher-level policy and provides feedback. The cumulative cost, followed by the high-level strategy continuing to choose the next... .

5. The charging pile positioning method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The second subtask in S2, rotating the robot to face the charging station, includes: The subtask The rotating robot faces the charging station. Considering only the angle error, the cumulative cost of the entire rotation process is defined as the cumulative cost of the rotation angle error: ； in, This indicates the angle deviation at the current moment, determined based on the position of the final target point in the robot's base coordinate system. Indicates the weighting coefficient. Indicates the angle at the current moment; Further minimize the cost at the current time step using the policy network. To output angular velocity ;in, Indicates the proportionality coefficient; When the deviation is within the tolerance range, a signal is sent to the higher-level policy to complete the current option.

6. The charging pile positioning method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The third subtask in S2, line tracking based on reflector data, includes: After the robot rotates to the correct orientation, linear tracking is performed, and the desired rotation angle is calculated according to the Stanley algorithm. : ; in, This represents the heading error, which is the difference between the robot's current heading angle and the desired heading angle. Indicates lateral error. This represents the proportionality coefficient, used to adjust the strength of the correction for lateral errors. Indicates the robot's moving speed. Represents a constant; Define the current Cost function: ; in, , , Indicates the weighting coefficient. This indicates the robot's maximum speed during movement; Furthermore, the cost function is minimized using gradient descent through the built-in optimizer, the linear velocity is iteratively updated, and the desired steering angle is converted into angular velocity based on the robot's wheelbase L. The specific calculation formula is as follows: ； ； in, This represents the learning rate, which controls the step size of gradient descent. This represents the partial derivative of the cost function with respect to the linear velocity at the current moment. This represents the damping coefficient, used to reduce high-frequency jitter in angular velocity.

7. The charging pile positioning method based on hierarchical reinforcement learning as described in claim 1, characterized in that, The fourth subtask in S2, rotating the robot to align the electrodes with the charging station and then back to the final target point, includes: As the final execution link of the underlying policy network, this It will perform two consecutive actions: rotation and backward movement. Closed-loop control is achieved through a hierarchical reinforcement learning policy network. The cost functions for the two independent actions and the final total cost function are as follows: ； ； ； in, This represents the action weighting coefficient, used to distinguish the priority of rotation and backward movement. At that time, only the rotation cost is optimized. When the value is 0, only the backoff cost is optimized; , , , Indicates the proportionality coefficient; The angle and distance deviations calculated based on LiDAR or visual data are used as inputs to the underlying policy network. The network output layer then minimizes the total cost function. Generate control variables; The underlying network first predicts a temporary angular velocity. Calculate its rotation cost The gradient of the cost with respect to the predicted value is calculated using backpropagation. The final output is the angular velocity corrected by gradient descent: ； in, The learning rate is used to ensure Adjust in the direction of least cost; Once the rotation is complete, the motion weighting coefficient... When the value is changed to 1, the network predicts a temporary linear velocity. Calculate the backoff cost gradient of cost with respect to predicted value Output the corrected linear velocity: ； When the underlying policy network detects that the last option meets the termination condition, it determines that the entire process of positioning is complete. At this point, the upper-layer policy network records the total cost and task completion status of this localization, triggering the global update mechanism of reinforcement learning: adjusting the weights of the upper-layer policy based on the deviation between the total cost and the preset optimal threshold, and simultaneously... The local optimization results are fed back to the parameter update process of the underlying policy network, thereby optimizing the entire positioning system.

Images