A reinforcement learning-based inspection robot nearest point search and navigation method

Abstract

This invention provides a reinforcement learning-based method for finding the nearest point and navigating a patrol robot, comprising: Step 1: Setting two routes A and B that the patrol robot needs to inspect; Step 2: Obtaining the initial position coordinates of the current patrol robot and activating the LiDAR; Step 3: Obtaining the current yaw angle of the patrol robot, allowing the robot to move straight for m seconds, and obtaining the current position coordinates of the patrol robot, unifying the coordinate system; Step 4: Selecting the nearest path point as the initial point for tracking; Step 5: Connecting the initial point with the next path point to obtain the angle relative to the IMU coordinate system, controlling the point-to-point movement of the patrol robot; Step 6: When patrolling route A is completed, starting patrolling route B, using a reinforcement learning algorithm to select the nearest path point to route B as the initial point for patrolling. This invention's method has high efficiency and accuracy, significantly reducing the computational complexity of existing technologies.

Description

Technical Field

[0001] This invention belongs to the field of computer technology, and in particular relates to a method for nearest point search and navigation of inspection robots based on reinforcement learning. Background Technology

[0002] With the accelerating pace of industrialization, a large number of robotic technologies are gradually replacing manual labor, and robots are increasingly used in security and safety functions to maintain social stability and protect public safety. Intelligent inspection robots have a wide range of applications, including data centers, industrial parks, chemical plants, and substations. In RTK-based inspection robot navigation, the inspection route is not fixed. In the next inspection task, the vehicle's initial position may not be a waypoint on the route; it is necessary to find the nearest waypoint to the vehicle's current position and use this waypoint as the starting point for tracking. If obstacles are encountered during route changes, obstacle avoidance is required, and the nearest point after obstacle avoidance must be adaptively selected. This involves the problem of finding the nearest neighbors of a given geographical location.

[0003] Currently, the main methods for finding the nearest station based on geographic location are: full network traversal and grid partitioning. Full network traversal calculates the distance between the input location and all stations in the network; the station with the smallest distance is the nearest station. The number of distance calculations equals the number of stations in the network. Full network traversal requires calculating the distance between the input point and all stations in the network. If the number of stations is N, the algorithm's complexity is linear O(N), making it inefficient. Grid partitioning divides the network into several grids, each containing several stations. It first calculates the grid to which the input point belongs and its neighboring grids. Then, within the input and neighboring grids, it applies full network traversal to find the nearest station. The number of distance calculations equals the number of stations in the input and neighboring grids. The performance of grid partitioning depends on the size of the grid and is not universally applicable. Even within the input and neighboring grids, it still requires traversing all stations, resulting in low efficiency. Because the number of distance calculations in grid partitioning equals the number of stations in the input and neighboring grids, its algorithm complexity remains linear O(N).

[0004] Therefore, how to provide a location-based nearest point search method that reduces the computational complexity of existing technologies, improves work efficiency, saves computation time, and has universal applicability has become a technical problem that needs to be solved.

[0005] Reinforcement learning is used to describe and solve problems where an agent learns strategies to maximize rewards or achieve specific goals during interactions with its environment. If an agent's behavioral strategy leads to a positive reward (reinforcement signal) from the environment, the agent's tendency to adopt that strategy in the future will be strengthened. The agent's goal is to find the optimal strategy in each discrete state to maximize the expected discounted reward. Combining reinforcement learning with inspection robots allows the robot to find the shortest path in a shorter time, thereby improving inspection efficiency. Summary of the Invention

[0006] Purpose of the invention: The technical problem to be solved by the present invention is to address the shortcomings of the existing technology by providing a method for nearest point search and navigation of inspection robots based on reinforcement learning, including the following steps:

[0007] Step 1: Set the inspection robot to inspect two routes, A and B;

[0008] Step 2: Obtain the initial position coordinates (x1, y1) of the current inspection robot through the vehicle-mounted GNSS-RTK (Global Navigation Satellite System, also known as GNSS; RTK stands for Real-time kinematic, which refers to real-time dynamic positioning technology based on carrier phase observations), and turn on the lidar;

[0009] Step 3: The vehicle-mounted IMU (Inertial Measurement Unit) obtains the current yaw angle of the inspection robot. The inspection robot moves straight for m seconds (m ranges from 5 to 10), and obtains the position coordinates (x2, y2) of the inspection robot at this time. The angle difference between the RTK and IMU coordinate systems is calculated as actan((y2-y1) / (x2-x1)), thereby unifying the coordinate systems.

[0010] Step 4: Use a reinforcement learning algorithm to select the nearest path point as the initial point for tracking;

[0011] Step 5: Using the PID (Proportion Integral Derivative) algorithm, connect the initial point to the next path point to obtain the angle relative to the IMU coordinate system. Use the angle as the control variable of the PID to control the point-to-point movement of the inspection robot.

[0012] Step 6: When the inspection of route A is completed, start the inspection of route B. Use the reinforcement learning algorithm to select the shortest path point to route B as the initial point of the inspection.

[0013] In step 1, the remote-controlled inspection robot traverses routes A and B respectively, and records the robot's latitude and longitude (L1,A1), (L2,A2), (L3,A3), ..., (L...) along route A using onboard RTK. i A i ), the latitude and longitude of route B are (C1,D1), (C2,D2), (C3,D3), ..., (C j D j ), where L i A i Let C represent the longitude and latitude of the i-th point in route A, respectively. j D j Let these represent the longitude and latitude of the j-th point in route B, respectively.

[0014] Using the Gauss-Kruger projection, the latitude and longitude (L1,A1), (L2,A2), (L3,A3), ..., (L i A i Convert the coordinates of the x1, y1, (x2, y2), (x3, y3), ..., (x2, y2) into a Cartesian coordinate system. i ,y i ), where (x i ,y i ) is (L i A i Convert the coordinates to a Cartesian coordinate system.

[0015] Using the Gauss-Kruger projection, the latitude and longitude (C1,D1), (C2,D2), (C3,D3), ..., (C j D j Convert the coordinates of the g1, h1, g2, h2, g3, h3, ..., g3 to a Cartesian coordinate system. j ,h j ), of which (g j ,h j ) is (C j D j Convert the coordinates to a Cartesian coordinate system.

[0016] The coordinates are then stored in the set Z.

[0017] Step 4 includes:

[0018] Step 4-1: Create and initialize an empty database D with a capacity of N to store the data and trajectories during the optimization process;

[0019] Step 4-2, set the number of training iterations to H;

[0020] Step 4-3: Establish and initialize the neural network Q, such as the neural network Q in Deep Q Network Learning (DQN), to approximate the action value function of the currently selected nearest path point, and randomly generate the parameters w1 of the neural network;

[0021] Step 4-4: Establish and initialize another neural network T with the same structure as Q, which is used to generate target action value as the basis for optimizing the current neural network Q. Set the same parameters as Q: w2 = w1.

[0022] Steps 4-5: Use the vehicle-mounted RTK to obtain the current state value (S) of the inspection robot. t S t Indicates the current position;

[0023] Steps 4-6: Discretize the nearest path point approximation process into M equally spaced time points using time intervals τ;

[0024] Steps 4-7: Return to step 4-5 and continue the training process until H training tasks are completed.

[0025] Steps 4-6 include:

[0026] Step 4-6-1: Use LiDAR to collect point cloud data of the surrounding environment. Points with distance data returned by the LiDAR less than or equal to N1 (generally 5) meters are identified as obstacle point clouds. Cluster the obstacle point clouds and calculate the boundary of the area that the inspection robot can safely pass through based on the obtained clustering results.

[0027] Step 4-6-2: Obtain the left yaw angle θ′1 and right yaw angle θ′2 relative to the vehicle position through the boundary of the obstacle range, and calculate the current position S respectively. t The yaw angles θ1, θ2, ..., θ relative to the path coordinates are as follows: i θ i Indicates the current position S t The yaw angle relative to the i-th coordinate point on the path;

[0028] Let θ1, θ2, ..., θ i Compare with θ′1 and θ′2 respectively, and... The path points are placed into the constraint set E. If the lidar does not detect any obstacles, then E = Z.

[0029] Step 4-6-3: Use a greedy strategy to search and generate the currently selected nearest path point a in constraint set B. t t takes values ​​from 1 to M;

[0030] Step 4-6-4: Use the PID algorithm to set the current position S. t With the nearest path point a tConnecting the points, we obtain the angle relative to the IMU coordinate system. This angle is used as the control variable for the PID, controlling the inspection robot's point-to-point movement for a time period τ. The reward value received is r. t ,r t r is the reciprocal of the Euclidean distance between the current position and the selected nearest path point. t =(‖S t -a t ‖) -1 And receive the new value of the location status state via the vehicle-mounted RTK, which is S. t+1 ;

[0031] Steps 4-6-5: Collect historical data (S) t a t r t S t+1 Store it in database D;

[0032] Step 4-6-6: Randomly select a sample d from D. j (S j a j r j S j+1 ), where j takes values ​​from 1 to M, and j ≠ t;

[0033] Steps 4-6-7, d j Inputting the neural network Q generates the current value Q(S) j ,a j ,w1), Q is S j ,a j A function of w1;

[0034] Steps 4-6-8, simultaneously d j The target value y is generated by inputting a neural network T. j This is used as the optimization of the target supervised neural network Q;

[0035] Steps 4-6-9, calculate y j With Q(S) j ,a j Find the gradient of Δj, which is the difference between w1 and w2.

[0036] Steps 4-6-10: Update parameter w1 using gradient descent. Where α is the learning rate;

[0037] Steps 4-6-11: Update the neural network T parameters every C steps, let w2 = w1, and C be a positive real number.

[0038] This invention addresses the problem of finding initial path points and finding the shortest path points when changing routes in inspection robots. It utilizes GNSS-RTK, IMU, and LiDAR sensors combined with reinforcement learning to construct a deep reinforcement learning framework. By training a shortest-point search strategy using deep reinforcement learning, it proposes an obstacle avoidance search strategy that uses LiDAR to remove occlusion points from the sampled data. This strategy has high efficiency and accuracy, and boasts advantages such as wide applicability, good search performance, short search time, and strong universality.

[0039] Beneficial effects: The method of the present invention has high efficiency and accuracy, significantly reduces the computational complexity of existing technologies, improves work efficiency, and is robust in environments with dense obstacles, making it universally applicable. Attached Figure Description

[0040] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments, and the advantages of the present invention in the above and / or other aspects will become clearer.

[0041] Figure 1 This represents the reward between the inspection robot and the waypoint (i.e., (Diagram)

[0042] Figure 2 This is a schematic diagram of the search space after removing the path points within the angle range from the inspection robot to the obstacle when the inspection robot detects an obstacle.

[0043] Figure 3 This is a schematic diagram of the path selection after obstacle avoidance.

[0044] Figure 4 This is a flowchart of the method of the present invention. Detailed Implementation

[0045] In this detailed implementation, a specific embodiment is provided, and the invention is illustrated with specific examples and parameter values ​​in conjunction with the accompanying drawings:

[0046] like Figure 4 As shown, the method of the present invention specifically includes:

[0047] The inspection robot is set to inspect two routes, A and B. The remotely controlled inspection robot traverses both routes A and B. The vehicle-mounted RTK records the robot's latitude and longitude (L1, A1), (L2, A2), (L3, A3), ..., (L...) along route A. i A i ), the latitude and longitude of route B are (C1,D1), (C2,D2), (C3,D3), ..., (C j D j ), where L i Ai Let C represent the longitude and latitude of the i-th point in route A, respectively. j D j Let these represent the longitude and latitude of the j-th point in route B, respectively.

[0048] Using the Gauss-Kruger projection, the latitude and longitude (L1,A1), (L2,A2), (L3,A3), ..., (L i A i Convert the coordinates of the x1, y1, (x2, y2), (x3, y3), ..., (x2, y2) into a Cartesian coordinate system. i ,y i ), where (x i ,y i ) is (L i A i Convert the coordinates to a Cartesian coordinate system.

[0049] Using the Gauss-Kruger projection, the latitude and longitude (C1,D1), (C2,D2), (C3,D3), ..., (C j D j Convert the coordinates of the g1, h1, g2, h2, g3, h3, ..., g3 to a Cartesian coordinate system. j ,h j ), of which (g j ,h j ) is (C j D j Convert the coordinates to Cartesian coordinates; store the coordinates in set Z.

[0050] The initial position coordinates (x1, y1) of the inspection robot are obtained through vehicle-mounted GNSS-RTK (RTK, Real-time kinematic, real-time dynamic positioning technology based on carrier phase observations), and the lidar is activated. The vehicle-mounted IMU (Inertial Measurement Unit) obtains the current yaw angle of the inspection robot. The inspection robot moves straight for ms (m ranges from 5 to 10 s) to obtain the position coordinates (x2, y2) of the inspection robot at this time. The angle difference between the RTK and IMU coordinate systems is calculated using the formula actan((y2-y1) / (x2-x1)), thereby unifying the coordinate systems.

[0051] The reinforcement learning algorithm is used to select the shortest path point as the initial point for tracking. An empty database D with a capacity of N is established and initialized to store the data and trajectory during the optimization process. A neural network Q is established and initialized, such as the neural network Q in Deep Q-Network Learning (DQN), to approximate the action value function of the currently selected shortest path point. The parameters w1 of this neural network are randomly generated. Another neural network T with the same structure as Q is established and initialized to generate the target action value as the basis for the current neural network Q optimization. The same parameters w2 = w1 are set as Q. The current state S of the inspection robot is obtained using onboard RTK. t S t Indicates the current position;

[0052] The process of approximating the nearest path point is discretized into M equally spaced time points by time interval τ. The surrounding environment point cloud data is collected by LiDAR. If the distance data returned by the LiDAR is less than or equal to 5 meters, it is considered an obstacle point cloud. The obstacle point cloud is clustered, and the obtained clustering results are used to calculate the boundary of the area that the inspection robot can safely pass through.

[0053] By obtaining the left yaw angle θ′1 and right yaw angle θ′2 relative to the vehicle's position through the boundary of the obstacle range, the yaw angles θ1, θ2, ..., θ′2 relative to the current state St and the path coordinates are calculated respectively. i , θ1,θ2…,θ i Compare with θ′1 and θ′2 respectively, and... The path points are placed into the constraint set E. If the lidar does not detect any obstacles, then E = Z. A greedy strategy is used to search and generate the currently selected nearest path point a in the constraint set B. t t takes values ​​from 1 to M. For example... Figure 2 , Figure 3 As shown.

[0054] Using the PID algorithm, the current position S t With the nearest path point a t Connecting the points gives the angle relative to the IMU coordinate system. This angle is used as the control variable for the PID, controlling the inspection robot's point-to-point movement for a time τ. For example... Figure 1 As shown, the value of the reward received is r. t ,r t r is the reciprocal of the Euclidean distance between the current position and the selected nearest path point. t =(‖S t -a t ‖)-1 And receive the new value of the position status state via the vehicle-mounted RTK, which is S. t+1 Collect historical data (S t a t r t S t+1 Store the sample d in database D, and randomly select a sample d from D. j (S j a j r j S j+1 ), where j takes values ​​from 1 to M, and j ≠ t;

[0055] d j Inputting the neural network Q generates the current value Q(S) j ,a j ,w1), Q is S j ,a j A function of w1;

[0056] At the same time, d j The target value y is generated by inputting a neural network T. j This is used as the optimization of the target supervised neural network Q; y is calculated. j With Q(S) j ,a j Find the gradient of Δj, which is the difference between w1 and w2.

[0057] The parameter w1 is updated using gradient descent. Where α is the learning rate.

[0058] Every C steps (C is a positive real number), update the parameters of the neural network T, setting w2 = w1. Then, return to the current state S of the inspection robot obtained using the onboard RTK. t Then continue training until the training task is completed H times (H is 500 in this embodiment).

[0059] When the inspection of route A is completed, the inspection of route B begins. A reinforcement learning algorithm is used to select the path point closest to route B as the initial point for inspection.

[0060] In its specific implementation, this application provides a computer storage medium and a corresponding data processing unit. The computer storage medium is capable of storing a computer program, which, when executed by the data processing unit, can run the invention's content regarding a reinforcement learning-based nearest-point search and navigation method for inspection robots, as well as some or all of the steps in various embodiments. The storage medium can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc.

[0061] Those skilled in the art will clearly understand that the technical solutions in the embodiments of the present invention can be implemented using computer programs and their corresponding general-purpose hardware platforms. Based on this understanding, the technical solutions in the embodiments of the present invention, or the parts that contribute to the prior art, can be embodied in the form of computer programs, i.e., software products. These computer program software products can be stored in a storage medium and include several instructions to cause a device containing a data processing unit (which may be a personal computer, server, microcontroller, MUU, or network device, etc.) to execute the methods described in various embodiments or certain parts of the embodiments of the present invention.

[0062] This invention provides a method for nearest point search and navigation of an inspection robot based on reinforcement learning. Many methods and approaches exist for implementing this technical solution; the above description is merely a preferred embodiment of the invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of this invention, and these improvements and modifications should also be considered within the scope of protection of this invention. All components not explicitly stated in this embodiment can be implemented using existing technologies.

Claims

1. A method for nearest point search and navigation of an inspection robot based on reinforcement learning, characterized in that, Includes the following steps: Step 1: Set the inspection robot to inspect two routes, A and B; Step 2: Obtain the initial position coordinates (x1, y1) of the current inspection robot using the vehicle-mounted GNSS-RTK and turn on the LiDAR; Step 3: The vehicle-mounted IMU inertial measurement unit obtains the current yaw angle of the inspection robot. The inspection robot moves straight for m seconds and obtains the position coordinates (x2, y2) of the inspection robot at this time. The angle difference between the RTK and IMU coordinate systems is calculated as arctan((y2-y1) / (x2-x1)), thereby unifying the coordinate systems. Step 4: Use a reinforcement learning algorithm to select the nearest path point as the initial point for tracking; Step 4 includes: Step 4-1: Create and initialize an empty database D with a capacity of N to store the data and trajectories during the optimization process; Step 4-2, set the number of training iterations to H; Step 4-3: Establish and initialize the neural network Q to approximate the action value function of the currently selected nearest path point, and randomly generate the parameters w1 of the neural network; Step 4-4: Establish and initialize another neural network T with the same structure as Q, which is used to generate target action value as the basis for optimizing the current neural network Q. Set the same parameters as Q: w2=w1. Steps 4-5: Use the vehicle-mounted RTK to obtain the current state value (S) of the inspection robot. t S t Indicates the current position; Steps 4-6 involve approximating the nearest path point over time intervals. Discretized into M equally spaced time points; Steps 4-7: Return to step 4-5 and continue the training process until H training tasks are completed; Step 4-6-1: Use LiDAR to collect point cloud data of the surrounding environment. Points with distance data returned by the LiDAR less than or equal to N1 meters are identified as obstacle point clouds. Cluster the obstacle point clouds and calculate the boundary of the area that the inspection robot can safely pass through based on the obtained clustering results. Step 4-6-2: Obtain the left yaw angle relative to the vehicle's position by using the boundary of the obstacle range. yaw angle to the right Calculate the current position S respectively t Yaw angle relative to path coordinates , Indicates the current position S t The yaw angle relative to the k-th coordinate point on the path; Will respectively with , Compare and , The path points are placed into the constraint set E. If the lidar does not detect an obstacle, then... ; Step 4-6-3, using a greedy strategy on the constraint set Search and generate the nearest path point for the currently selected path. t takes values ​​from 1 to M; Step 4-6-4: Use the PID algorithm to set the current position S. t With the nearest path point Connecting the points, we obtain the angle relative to the IMU coordinate system. This angle is used as the control variable for the PID controller, controlling the inspection robot to move point-to-point for a certain period of time. The value of the reward received is r. t , r t r is the reciprocal of the Euclidean distance between the current position and the selected nearest path point. t = And receive the new value of the location status state via the vehicle-mounted RTK, which is S. t+1 ; Steps 4-6-5: Collect historical data ( , r t S t+1 Store it in database D; Step 4-6-6: Randomly select a sample d from database D. j (S) j , r j S j+1 j takes values ​​from 1 to M, and j ≠ t; Steps 4-6-7, sample d j (S) j , r j S j+1 The input to the neural network Q generates the current value. Q is The function; Steps 4-6-8, simultaneously sample d j (S) j , r j S j+1 The input neural network T generates the target value y. j This is used as the optimization of the target supervised neural network Q; Steps 4-6-9, calculate y j and The difference Solve gradient ); Steps 4-6-10: Update the parameter w1 using gradient descent. ),in The learning rate; Steps 4-6-11: Update the neural network T parameters every C steps, let w2=w1, where C is a positive real number; Step 5: Using the PID algorithm, connect the initial point with the next path point to obtain the angle relative to the IMU coordinate system. Use the angle as the control variable of the PID to control the point-to-point movement of the inspection robot. Step 6: When the inspection of route A is completed, start the inspection of route B. Use the reinforcement learning algorithm to select the shortest path point to route B as the initial point of the inspection.

2. The method according to claim 1, characterized in that, In step 1, the remote-controlled inspection robot traverses routes A and B respectively, and records the robot's latitude and longitude (L1,A1), (L2,A2), (L3,A3), ..., (L...) along route A using onboard RTK. i A i ), the latitude and longitude of route B (C1,D1), (C2,D2), (C3,D3),..., (C j D j ), where L i A i Let C represent the longitude and latitude of the i-th point in route A, respectively. j D j These represent the longitude and latitude of the j-th point in route B, respectively.

3. The method according to claim 2, characterized in that, In step 1, the Gauss-Kruger projection is used to map the latitude and longitude (L1,A1), (L2,A2), (L3,A3), ..., (L... i A i Convert the x1, y1, (x2, y2), (x3, y3), ..., (x2, y2) coordinates into a Cartesian coordinate system. i , y i ), where (x i , y i ) is (L i A i Convert to coordinates in a Cartesian coordinate system; Using the Gauss-Kruger projection, the latitude and longitude (C1,D1), (C2,D2), (C3,D3), ..., (C j D j Convert the coordinates of the given coordinates to (g1, h1), (g2, h2), (g3, h3), ..., (g...) in a Cartesian coordinate system. j , h j ), of which (g j , h j ) is (C j D j Convert the coordinates to a Cartesian coordinate system. Then store the coordinates in a set. middle.

Images