Intelligent robot path planning method based on visual identification

Abstract

The invention relates to the technical field of intelligent robot path planning based on visual identification, and discloses an intelligent robot path planning method based on visual identification. According to the scheme, the problems of inconsistency of environment sensing data and lack of accurate geometric information are solved by utilizing data acquisition and calibration processing of the camera and the laser radar. The camera collects images, and parameters are determined by hardware; the internal reference matrix of the camera ensures clear correspondence between pixels and actual coordinates. The laser radar collects point cloud data, records three-dimensional coordinates of the environment, and is high in precision and real-time. Through data fusion and calibration, point cloud and image alignment by the system, camera internal reference matrix establishment and external reference calibration, accurate mapping and back projection are realized, and grassland area and obstacle information is extracted. According to the method, the sensing precision is improved, support is provided for path planning and navigation, and the robot has accurate positioning and efficient path planning capabilities.

Description

Technical Field

[0001] The present invention relates to the technical field of path planning for intelligent robots based on visual recognition, and specifically to a method for path planning of intelligent robots based on visual recognition. Background Art

[0002] In the field of intelligent robots, path planning methods based on visual recognition and lidar technology have been widely applied in many fields such as agriculture, gardening, and autonomous driving. Existing technologies usually use cameras to obtain environmental images, and combine the point cloud data collected by lidar. Through sensor calibration and data fusion, the recognition and segmentation of targets such as grasslands and obstacles in the working area are realized. Common image processing methods include color histogram analysis, threshold segmentation, and connected component analysis to determine the grassland area; at the same time, lidar data detects protruding obstacles by calculating the height difference between each point in the point cloud and the ground. On this basis, through the external parameter calibration technology, the three-dimensional point cloud data collected by lidar is converted into the camera coordinate system, and then mapped onto the image plane using the perspective projection theory, so that data from different sensors can be fused and calculated in the same coordinate system.

[0003] In existing methods, the back-projection technique is a commonly used means to map the grassland area segmented in the image from pixel coordinates to the actual world coordinates. It is usually assumed that the ground is flat, and the world coordinates corresponding to each image point are obtained using the camera's internal parameter matrix and external parameter matrix. Although this method is widely used in academia and industry, it often relies on accurate calibration results and assumption conditions. If the ground is not completely flat or there are errors in calibration, the mapping results will deviate. At the same time, the process of detecting obstacles by lidar usually relies on statistically analyzing the height data of the point cloud, and a height threshold is set to distinguish between the ground and non-ground objects. The detected obstacle point cloud needs to be mapped to convert it from the lidar coordinate system to the same world coordinate system as the grassland after back-projection. In existing technologies, a common practice is to use the external parameter matrix between the lidar and the camera to complete this conversion, and then construct an obstacle mask on a discrete grid. However, due to the uncertainty of the data collected by the sensor and the possible errors in the external parameter calibration process, the accuracy of the obstacle mask often cannot be fully guaranteed, which brings certain difficulties to subsequent path planning.

[0004] In terms of path planning, existing technologies mainly rely on graph search-based, sampling algorithms, and continuous path optimization methods based on gradient descent. These methods often need to introduce multiple weight parameters during the planning process to balance path length, smoothness, and obstacle avoidance requirements. However, most of these weight parameters rely on empirical data or repeated trials and lack direct physical basis, resulting in significant differences in the robustness and applicability of the algorithms in different environments. In addition, the implementation processes of existing methods for obstacle safety distance, local area center tracking, and path smoothness optimization are often described rather briefly, without giving clear mathematical calculation formulas and specific explanations of each variable, making it difficult to obtain accurate path planning results in practical applications. Moreover, during the initial path generation process, linear interpolation between the starting point and the ending point is usually adopted, but linear interpolation cannot reflect the true shape of the grassland in the actual working area and cannot fully consider the obstacle distribution, so subsequent adjustment and optimization are required. In existing technologies, the selection of the number of path points often depends on empirical values or preset expected intervals. Although this method is simple and easy to implement, it is difficult to ensure the fineness and planning accuracy of the path in different environments.

[0005] Based on the above background, the purpose of this case is to propose an intelligent robot path planning method based on visual recognition, which performs more refined processing on camera images and lidar data on the basis of existing technologies, ensuring that each link from grassland back-projection, obstacle mapping, initial path generation, obstacle safety adjustment, local center tracking adjustment, to path smoothness optimization can be achieved through directly measured physical parameters and definite mathematical expressions, thereby reducing the dependence on empirical free parameters. Summary of the Invention

[0006] The present invention provides an intelligent robot path planning method based on visual recognition, which promotes the solution of the problems mentioned in the above background technology.

[0007] The present invention provides the following technical solutions: An intelligent robot path planning method based on visual recognition, comprising:

[0008] Denote the image collected by the camera as I(u, v);

[0009] where u is the horizontal pixel coordinate of the image, with the value range [0, W - 1]; v is the vertical pixel coordinate of the image, with the value range [0, H - 1]; W is the image width; H is the image height;

[0010] Let the camera internal parameter matrix be K, specifically:

[0011]

[0012] where f x and f y are the focal lengths in the horizontal and vertical directions respectively; cx With c y Are the horizontal and vertical coordinates of the principal point of the image, respectively;

[0013] Let the point cloud set collected by the lidar be

[0014] Among them, P l,i Is the three-dimensional coordinate of the i-th point; X l,i , Y l,i And Z l,i Are the x, y, and z coordinates of the i-th point in the lidar coordinate system, respectively; N l Is the total number of point clouds.

[0015] Optionally, it also includes data synchronization and camera-lidar calibration, specifically:

[0016] Set the lidar-to-camera coordinate system conversion algorithm and convert the lidar points to the camera coordinate system, specifically:

[0017] Let P l,i =(X l,i , Y l,i , Z l,i ) T ;

[0018] Then convert to the camera coordinate system to get:

[0019] P c,i =R·P l,i +T, where P c,i =(X c,i , Y c,i , Z c,i ) T ;

[0020] Among them, Is the rotation matrix, and its elements are rotation coefficients; T=(T x , T y , T z ) T , is the translation vector, and the components T x , T y And T z Represent the translation amounts along the x, y, and z directions, respectively;

[0021] Among them, Is the set of real numbers; the rotation matrix R satisfies R T R = I, R T Is the transpose of R, I is the 3×3 identity matrix, and at the same time satisfies det(R)=1;

[0022] Set the perspective projection algorithm and project P c,i onto the image plane, specifically:

[0023] Among them, u i and v i are the pixel coordinates of the i-th point in the image; Z c,i is the depth of point P c,i ;

[0024] Given the calibrated corresponding point pairs {(P l,i , p i ) | i = 1, 2,..., N c};

[0025] Among them, p i = (u i , v i ) T , is the corresponding pixel coordinate in the image; N c is the number of calibrated point pairs;

[0026] Set the projection error function as:

[0027] Among them, the projection function Among them, P = (P x , P y , P z ) T ;

[0028] To ensure that R is orthogonal, it is represented by Euler angles:

[0029] R = R x (α)R y (β)R z (γ); where α is the rotation angle around the x-axis, β is the rotation angle around the y-axis, and γ is the rotation angle around the z-axis; R x (α), R y (β) and R z (γ) are the matrices around the x-axis, y-axis, and z-axis respectively, specifically:

[0030]

[0031] The parameter vector is denoted as θ = (α, β, γ, T x , T y , T z ) T ;

[0032] Update the parameters using the gradient descent method, specifically as follows:

[0033]

[0034] Among them, k is the number of iterations; η > 0, is the step size; is the gradient of the error function E with respect to θ, and its j-th component is:

[0035] where θ j is the j-th element of the vector θ.

[0036] Optionally, it includes grassland area segmentation, specifically:

[0037] For each pixel I(u, v) in the image, whose color components R(u, v), G(u, v), B(u, v) all belong to [0, 255], set the judgment function:

[0038]

[0039] where Γ outputs a binary value, 1 represents the grassland area, and 0 represents the non-grassland area;

[0040] Obtain the grassland mask: M g (u, v) = Γ(R(u, v), G(u, v), B(u, v)).

[0041] Optionally, it also includes obstacle detection, specifically:

[0042] Using the lidar point cloud data, calculate the 10th percentile of all Z l,i values to obtain the ground height:

[0043]

[0044] where Percentile(*, 10%) represents the 10th percentile in the set;

[0045] For each point P l,i = (X l,i , Y l,i , Z l,i ) set the height difference: h i = Z l,i - Z ground ;

[0046] where h i is the height difference of the i-th point relative to the ground;

[0047] Set the obstacle height threshold, denoted as h th , and set the judgment function:

[0048]

[0049] where Δ(h i ) = 1 indicates that this point is an obstacle;

[0050] Generate the obstacle mask: M o(X l,i , Y l,i ) = Δ(h i ).

[0051] Optionally, it further includes path planning, specifically:

[0052] Denote the grassland area obtained by back-projecting the image as

[0053] where X, Y are ground coordinates; 1 indicates that this position is a grassland area, and 0 indicates a non-grassland area;

[0054] Meanwhile, map the obstacles detected by the lidar to obtain an obstacle mask, denoted as M o (X, Y);

[0055] where M o (X, Y) = 1 indicates that there is an obstacle at this position, and 0 indicates no obstacle;

[0056] Set the feasible mowing area as:

[0057] Discretize the feasible area into grids with a fixed grid resolution r. Let the grid set be:

[0058]

[0059] where X p,q , Y p,q are the coordinates of the center of the (p, q)-th grid respectively; N x , N y are the number of grids in the horizontal and vertical directions respectively;

[0060] Let the starting point of the mowing robot be P s = (X s , Y s ) and the target end point be P e = (X e , Y e );

[0061] Obtain the straight-line distance between the starting point and the end point, denoted as L0;

[0062] Let the distance between the midpoints of the expected path be d s ;

[0063] Then the number of initial path points N is: rounded up;

[0064] Set the linear interpolation algorithm and generate the initial path, specifically:

[0065]

[0066] Among them, is the starting point, is the end point, and the remaining points are evenly distributed interpolation points.

[0067] Optionally, it also includes obstacle safety constraint adjustment, specifically:

[0068] S32. Obstacle safety constraint adjustment:

[0069] For each intermediate point P i =(X i , Y i ), perform the following steps:

[0070] S321. Calculate the distance to the nearest obstacle:

[0071] For point P i , calculate its Euclidean distance to all obstacle grids and take the minimum value:

[0072]

[0073] Among them, is all obstacle grids that satisfy M o (X, Y)=1; X G , Y G are the central coordinates of the obstacle grid G;

[0074] S322. Judge safety and calculate the adjustment vector:

[0075] Let the safety distance be d min ;

[0076] If D obs (P i )<d min , then adjustment is required, and the specific adjustment is:

[0077] Let G * be the obstacle grid that makes the distance reach the minimum value, that is:

[0078]

[0079] Set the unit vector pointing from the obstacle to P i as:

[0080]

[0081] Calculate the adjustment vector:

[0082] Update the point position: P i ←P i +ΔPi ;

[0083] S323. Iterative adjustment:

[0084] Repeat steps S321 and S322 until the corresponding point P i satisfies D obs (P i ) ≥ d min .

[0085] Optionally, it also includes local center tracking adjustment, specifically:

[0086] S33. Local center tracking adjustment:

[0087] For each intermediate point P i =(X i , Y i ), perform the following steps:

[0088] S331. Determine the local neighborhood:

[0089] Construct a square neighborhood centered at point P i with side length l around point P i . Denote the set of grids included in this neighborhood as:

[0090]

[0091] S332. Calculate the local centroid:

[0092] Calculate the average value of the center points of all feasible grids in the neighborhood:

[0093]

[0094] where, is the number of grids in the set ; X G , Y G are the coordinates of the center of each grid;

[0095] S333. Calculate the deviation and gradient:

[0096] Set the deviation distance as:

[0097] Calculate the gradient of this distance with respect to P i , and the direction is from P c,i to P i :

[0098]

[0099] S334. Update the point position:

[0100] Update point position:

[0101] Repeat steps S331 to S334 until the displacement between two adjacent updates is less than 0.005 m, and D is still satisfied after the update obs (P i )≥d min , if D is not satisfied obs (P i )≥d min , then return to step S32 first for safety adjustment.

[0102] Optionally, it also includes path smoothness optimization, specifically:

[0103] S34. Path smoothness optimization:

[0104] For three consecutive points P i-1 =(X i-1 ,Y i-1 ), P i =(X i ,Y i ), and P i+1 =(X i+1 ,Y i+1 ), perform the following steps:

[0105] S341. Calculate the included angle:

[0106] Calculate the vector:

[0107]

[0108] Then the included angle calculation is:

[0109]

[0110] Where, is the Euclidean norm of the vector ; is the Euclidean norm of the vector ;

[0111] S342. Smoothness judgment and update:

[0112] Obtain the minimum turning radius of the lawn mowing robot, denoted as R min ;

[0113] Set the maximum allowable included angle as φ max ,

[0114] Where, Δs is the average distance between adjacent points in the path;

[0115] If there exist three consecutive points that satisfy: φ(P i-1 ,Pi , P i+1 ) > φ max , then the intermediate point is directly updated smoothly:

[0116] S343. Iterative smoothing:

[0117] Steps S341 to S342 are sequentially executed for all consecutive three points in the path until all φ(P i-1 , P i , P i+1 ) in the entire path are not greater than φ max .

[0118] The present invention has the following beneficial effects:

[0119] 1. By collecting and calibrating the data of the camera and lidar in this solution, the problems of inconsistent environmental perception data and lack of accurate geometric information are solved, and there are significant beneficial effects. Specifically, first, image data is collected by the camera, where the horizontal pixel coordinates and vertical pixel coordinates in the image are determined by pixel values respectively, and the image width and height are directly given by the camera hardware parameters and resolution. Next, by determining the camera internal parameter matrix, which is composed of the focal lengths in the horizontal and vertical directions and the horizontal and vertical coordinates of the image principal point, and these parameters can be directly measured through camera calibration, so as to ensure the clear relationship between each pixel position in the image and the actual world coordinates. At the same time, the lidar collects point cloud data and records the three-dimensional coordinates of each point in the environment, and the coordinates of these points are directly collected by the lidar hardware, with high precision and real-time performance. In this way, the system obtains two sources of data: one is two-dimensional image data, and the other is three-dimensional point cloud data. Next, through data fusion and calibration processing, the three-dimensional point cloud data of the lidar can be effectively aligned with the image data of the camera, solving the problem of data inconsistency caused by different sensor perspectives and coordinate systems. This step of multi-sensor fusion, by establishing the camera internal parameter matrix and performing external parameter calibration, enables accurate mapping and back-projection between the image and the point cloud, so as to extract the accurate information of the grass area and obstacles in the environment. Finally, this method not only improves the accuracy of environmental perception, but also provides a solid geometric basis and data support for subsequent path planning and navigation decision-making. Through this series of steps, the system can seamlessly fuse two-dimensional images and three-dimensional lidar data, solve the problem of insufficient recognition caused by a single sensor being limited by the field of view, resolution or noise interference, and ensure that the lawn mowing robot can achieve precise positioning and efficient path planning in a complex environment.

[0120] 2. By setting up the laser radar to camera coordinate system conversion algorithm and using the perspective projection algorithm to convert the lidar point cloud data to the image plane, the problem of inconsistent coordinate systems between the camera and lidar data is solved, thus achieving high-precision fusion of multi-sensor data. Specifically, first, by setting the rotation matrix and translation vector, the point cloud data collected by the lidar is converted to the camera coordinate system, obtaining P c,i = R·P l,i + T. Subsequently, the converted point cloud data is mapped to the image plane through the perspective projection formula. Next, given the calibrated corresponding point pairs and constructing the projection error function, the parameters are iteratively updated by the gradient descent method to ensure that the calibration result makes the lidar data and image data accurately aligned in the same coordinate system. This series of steps solves the problems of coordinate system differences and data fusion difficulties caused by different sensor data, effectively maps the three-dimensional point cloud data to the two-dimensional image, and provides an accurate geometric basis for subsequent environment recognition and path planning. In addition, using Euler angles to represent the rotation matrix and optimizing the external parameters reduces the calibration error and improves the calibration accuracy and system robustness. Finally, this method realizes multi-sensor data synchronization, ensures that the robot can accurately identify the grass area and obstacles in a complex environment, and then plans a reasonable path. This not only improves the operation efficiency and navigation accuracy of the lawn mowing robot, but also provides a high-precision and low-error solution for similar multi-sensor fusion applications.

[0121] 3. By setting up a color judgment function for grassland area segmentation, the problem that it is difficult to accurately distinguish grassland from non-grassland areas in complex environments in traditional vision algorithms is solved. Specifically, in this step, the color components of each pixel in the image, namely the red, green, and blue values, are detected, and each pixel is classified using a pre-set threshold judgment condition as: grassland with an output value of 1 or non-grassland with an output value of 0. This binary processing enables the distribution of grassland in the image to be clearly extracted, forming a grassland mask. In this way, through the clear extraction of color features, the system can stably identify grassland areas under various lighting and background conditions, thus avoiding recognition errors caused by color interference or cluttered backgrounds. Specifically, through this step, first, the problem of inaccurate recognition caused by factors such as uneven lighting, color changes, or mixed backgrounds is solved; second, the binary grassland mask provides accurate area information for subsequent path planning, obstacle detection, and robot navigation, ensuring that the robot can focus on the grassland area during operation and will not stray into non-grassland areas; finally, this method simplifies the data processing flow and quickly generates a grassland mask through simple pixel-level judgment, thereby improving the overall response speed and real-time performance of the system. In summary, by using the color judgment function to segment each pixel in the image and accurately distinguish the grassland area from the non-grassland area, this step not only overcomes the problem of improper handling of complex backgrounds in traditional methods but also provides a clear and accurate environmental map for the subsequent tasks of intelligent lawn mowing robots, thus achieving efficient and accurate path planning and operation execution.

[0122] 4. Obstacle detection is achieved by calculating the ground height and setting a height difference threshold, which solves the problem of obstacle recognition errors caused by terrain undulation and noise in traditional methods. Specifically, in this step, the 10th percentile of all points is first calculated using lidar point cloud data as the ground height reference to avoid ground estimation deviation caused by extreme high or low values. Then, by calculating the height difference of each point and comparing it with the preset obstacle height threshold, it is accurately determined whether the point belongs to an obstacle. Finally, all the points determined to be obstacles together form an obstacle mask, providing reliable data for subsequent path planning and obstacle avoidance decision-making. Through the above steps, the misjudgment problem caused by terrain changes is first solved. Traditional methods may be affected by local high and low undulations, resulting in normal ground being misidentified as an obstacle or low obstacles being ignored. However, through the percentile statistical method, the interference of abnormal points can be reduced while ensuring accuracy, making the ground height calculation more stable. Secondly, this method effectively meets the detection requirements of objects at different heights and can adaptively adjust the obstacle detection standard. For example, in rugged terrain, the ground height will fluctuate, but since this method is based on statistical analysis, it will not easily misjudge normal terrain changes as obstacles, improving the robustness of detection. In addition, the generated obstacle mask can provide clear environmental perception data for intelligent devices, enabling them to accurately identify obstacles ahead and adjust the path in advance, thereby improving the obstacle avoidance ability and ensuring operation safety. Compared with traditional methods, this method is computationally efficient, can update obstacle information in real time, and is suitable for complex dynamic environments. Generally speaking, this step determines the ground height through statistical methods, combines height difference calculations to detect obstacles, improves detection accuracy, reduces misjudgment, and provides accurate data support for subsequent path optimization and obstacle avoidance strategies.

[0123] 5. Through grassland area back-projection, obstacle mapping, feasible area calculation, and grid path generation, precise path planning is achieved, solving the problems of poor adaptability to complex environments, ineffective obstacle avoidance, and inaccurate movement of traditional path planning for lawn mowing robots. Specifically, first, computer vision technology is used to back-project the camera image, converting the grassland area into the ground coordinate system to ensure accurate differentiation between feasible and non-feasible areas during path planning. Second, the obstacle data detected by the lidar is mapped to the same coordinate system, and an obstacle mask is generated to eliminate non-passable areas, ensuring that the robot's path does not overlap with obstacles. Then, by cross-computing the grassland area and the obstacle mask, the final feasible mowing area is determined, allowing passage only in grassland areas without obstacles, thereby improving the accuracy of path planning. Next, the feasible area is gridded, discretizing the ground in units of half of the robot's size, making the path points more regular and facilitating the planning of a smooth path that conforms to the robot's motion characteristics. During path generation, first, the straight-line distance between the starting point and the target point is calculated, and the path point interval is set. The initial path is generated using linear interpolation, making the path points evenly distributed to ensure that the robot moves along a reasonable trajectory. Compared with traditional methods, this solution can update the path in real time based on visual and lidar information to adapt to dynamic environmental changes. At the same time, the grid method makes path generation more systematic and reasonable, avoiding unnecessary twists or detours in the path and improving the robot's navigation efficiency in complex terrains. Finally, this method ensures that the lawn mowing robot can complete the work efficiently and safely, reduce ineffective travel, optimize the mowing coverage rate, and improve the overall efficiency of the mowing operation.

[0124] 6. By calculating the distance to the nearest obstacle, judging safety, adjusting path points, and iteratively optimizing the path to make the path comply with safety constraints, the problems of the path being too close to obstacles and insufficient driving safety that may occur in traditional path planning are solved, ensuring that the lawn mowing robot can operate safely and efficiently in a complex environment. First, in step S321, calculate the Euclidean distance from each intermediate path point to all obstacle grids and take the minimum value to measure the shortest distance between the path point and the obstacle. This calculation ensures that subsequent adjustments can be optimized for the nearest obstacle, preventing the robot from entering high-risk areas. Next, in step S322, set a safety distance threshold and judge whether the path point meets the safety requirements. If the distance from the path point to the obstacle is less than the safety distance, adjustment is required. To this end, first determine the obstacle grid closest to the path point and calculate the unit vector pointing from the obstacle to the path point to ensure that the adjustment direction is away from the obstacle. Subsequently, calculate the adjustment vector according to the unit vector and update the new position of the path point to offset it towards the safe area, thereby reducing the collision risk. In step S323, iteratively optimize the path to ensure that all path points meet the safety distance requirements. If a certain point still does not meet the safety constraints, repeat the above steps and continuously adjust its position until all path points are in the safe area. This iterative process guarantees the optimization of the path in terms of safety and rationality, avoiding excessive deviation of the path caused by insufficient or excessive adjustment at one time. Compared with traditional methods, this solution optimizes the path point by point to ensure that each path point is within the safe range, rather than simply globally offsetting the path, avoiding problems such as unnatural paths or excessive detours. In addition, through Euclidean distance calculation and unit vector adjustment, the path optimization is more refined, improving the robot's navigation ability in complex environments, ensuring that the path can effectively avoid obstacles and maintain the shortest possible driving distance, thereby improving the operation efficiency and reducing energy consumption. Finally, this method can significantly improve the safety and stability of the lawn mowing robot, enabling it to smoothly execute tasks in an environment with obstacles.

[0125] 7. By constructing a local neighborhood, calculating the local centroid, determining the adjustment direction, and iteratively optimizing the path points, the path is made smoother, solving the problem of path deviation or curvature mutation caused by obstacle avoidance adjustment, ensuring that the mowing robot travels as close as possible to the center of the feasible area on the premise of safety, and improving the continuity of the path and the operation efficiency. First, in step S331, for each intermediate path point, a square local neighborhood centered at this point is constructed, and its side length is determined by the robot size and the area resolution, ensuring a reasonable neighborhood range that can contain enough feasible area points for calculation without being too large to cause unstable optimization. By defining the neighborhood grid set, it is ensured that subsequent calculations are only based on the feasible area, improving the effectiveness of the adjustment. Then, in step S332, the average value of the center points of all feasible grids within this neighborhood, that is, the local centroid, is calculated as the ideal driving center position. This step ensures that the calculated centroid will not deviate due to a single abnormal point by averaging all feasible area points, and at the same time makes the path points as close as possible to the center of the feasible area, improving the rationality of the path. Next, in step S333, the deviation of the path point relative to the local centroid is calculated, and the direction of the deviation gradient is used to guide the adjustment direction. By calculating the unit vector from the current path point to the local centroid, it is ensured that the path adjustment direction always points to the optimal center of the feasible area instead of blindly deviating, improving the accuracy of the adjustment. In step S334, the path points are updated according to the calculated adjustment direction and iteratively optimized until the displacement between two adjacent adjustments is less than 0.005 m, ensuring smooth convergence of the path. If the updated path point exceeds the feasible area, return to step S32 for safety adjustment to ensure that the path not only meets local smoothness but also does not enter the obstacle area due to adjustment. Compared with the traditional method, this solution makes the path optimization more refined through local neighborhood calculation, avoiding unnecessary detours caused by obstacle avoidance adjustment, and improving the smoothness of the driving path and the operation efficiency. At the same time, by using the gradient optimization method, the path is gradually adjusted to the local optimum instead of a one-time jump adjustment, reducing the discontinuity of the path and making the robot travel more smoothly. In addition, this method combines safety constraint adjustment and local center tracking adjustment to ensure that the path can avoid obstacles and keep as close as possible to the center of the feasible area, improving the robot's autonomous navigation ability in complex environments, ultimately improving the operation efficiency and reducing energy consumption.

[0126] 8. By calculating the angles between path points, determining whether they exceed the smoothness threshold, and smoothly adjusting the points that exceed the threshold, the curvature change of the path becomes gentler, solving the sharp turn problem caused by uneven path point distribution or obstacle avoidance adjustment, and improving the feasibility of the path and the stability of the robot's travel. First, in step S341, for three consecutive points in the path, two adjacent vectors are calculated, and the angle is calculated using the vector angle formula to ensure that the local curvature of the path is measurable. This calculation method is based on the Euclidean distance, making the angle calculation stable and reliable, and capable of accurately reflecting the degree of path turning, providing data support for subsequent smoothing optimization. Then, in step S342, first obtain the minimum turning radius of the robot and calculate the maximum allowable angle to ensure that the path is within the robot's motion ability range, enabling the robot to smoothly pass through the path with the minimum turning radius. If it is found that the angle of a certain section of the path exceeds the allowable value, directly adjust the position of the middle point so that it falls on the line connecting the two end points to reduce the turning angle. After adjustment, it is necessary to check whether the updated point still meets the obstacle safety conditions and the local center tracking requirements to ensure that the optimization will not cause new safety hazards or lead to the path deviating from the optimal feasible area. Then, in step S343, traverse all consecutive three points, repeat the angle calculation and smoothing adjustment until the angles of all path points meet the constraint of the maximum allowable angle, thereby achieving global path smoothing optimization. This iterative method ensures that there are no sharp turns in the path, and the optimized path is smoother as a whole, improving the robot's autonomous driving ability in complex environments. Compared with traditional path optimization methods, this solution realizes path smoothness control by analyzing and optimizing the angles of local path points, enabling the robot to smoothly travel along the path and avoiding driving difficulties or out-of-control caused by sharp turns. At the same time, since the minimum turning radius of the robot's own kinematic constraints is combined in the optimization process, the feasibility of the path is ensured. In addition, this method can be dynamically adjusted and return to the previous obstacle safety constraint or local center tracking adjustment steps when necessary to ensure that the final path meets both the smoothness requirements and does not affect safety and path rationality, improving the stability and working efficiency of the robot in actual operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0127] Figure 1 It is a schematic flowchart of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0128] The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, rather than all of the embodiments. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present invention without creative efforts shall fall within the protection scope of the present invention.

[0129] Embodiment, an intelligent robot path planning method based on visual recognition, including:

[0130] Record the image collected by the camera as I(u, v);

[0131] Among them, u is the horizontal pixel coordinate of the image, with a value range of [0, W - 1], and the unit is pixel; v is the vertical pixel coordinate of the image, with a value range of [0, H - 1], and the unit is pixel; W is the width of the image, and the unit is pixel; H is the height of the image, and the unit is pixel; these values are determined by the camera hardware parameters and resolution;

[0132] Let the camera internal parameter matrix be K, specifically:

[0133]

[0134] Among them, f x and f y are the focal lengths in the horizontal and vertical directions respectively, with the unit of pixel, and can be directly measured by camera calibration; c x and c y are the horizontal and vertical coordinates of the principal point of the image respectively, with the unit of pixel, and can be directly measured by camera calibration;

[0135] Let the point cloud set collected by the lidar be

[0136] Among them, P l,i is the three-dimensional coordinate of the i-th point; X l,i , Y l,i and Z l,i are the x, y and z coordinates of the i-th point in the lidar coordinate system respectively, with the unit of meter; N l is the total number of point clouds; these values are directly collected by the lidar hardware.

[0137] Through the acquisition and calibration of camera and lidar data in this solution, the problems of inconsistent environmental perception data and lack of accurate geometric information are solved, which has significant beneficial effects. Specifically, first, image data is collected through the camera, where the horizontal and vertical pixel coordinates in the image are determined by pixel values respectively, and the width and height of the image are directly given by the camera hardware parameters and resolution. Next, by determining the camera intrinsic matrix, which is composed of the focal lengths in the horizontal and vertical directions and the horizontal and vertical coordinates of the image principal point, these parameters can be directly measured through camera calibration, thus ensuring the clear relationship between the position of each pixel in the image and the actual world coordinates. At the same time, the lidar collects point cloud data and records the three-dimensional coordinates of each point in the environment. The coordinates of these points are directly collected by the lidar hardware, with high precision and real-time performance. In this way, the system obtains two sources of data: one is two-dimensional image data, and the other is three-dimensional point cloud data. Next, through data fusion and calibration processing, the three-dimensional point cloud data of the lidar can be effectively aligned with the image data of the camera, solving the problem of data inconsistency caused by different sensor perspectives and coordinate systems. This multi-sensor fusion step, by establishing the camera intrinsic matrix and performing extrinsic calibration, enables accurate mapping and back-projection between the image and the point cloud, thereby extracting the accurate information of the grassland area and obstacles in the environment. Finally, this method not only improves the accuracy of environmental perception but also provides a solid geometric foundation and data support for subsequent path planning and navigation decision-making. Through this series of steps, the system can seamlessly fuse two-dimensional images and three-dimensional lidar data, solving the problem of insufficient recognition caused by a single sensor being limited by the field of view, resolution, or noise interference, and ensuring that the lawn mowing robot can achieve precise positioning and efficient path planning in complex environments.

[0138] It also includes data synchronization and camera-lidar calibration, specifically:

[0139] Set the lidar to camera coordinate system conversion algorithm and convert the lidar points to the camera coordinate system, specifically:

[0140] Let P l,i =(X l,i ,Y l,i ,Z l,i ) T ;

[0141] Then convert to the camera coordinate system to get:

[0142] P c,i =R·P l,i +T, where P c,i =(X c,i ,Y c,i ,Z c,i )T ;

[0143] Among them, is a rotation matrix, and its elements are rotation coefficients; T = (T x , T y , T z ) T , is a translation vector, and the components T x , T y and T z represent the translation amounts along the x, y, and z directions respectively, with the unit of meter;

[0144] Among them, is a set of real numbers; the rotation matrix R satisfies R T R = I, R T is the transpose of R, I is a 3×3 identity matrix, and at the same time satisfies det(R) = 1;

[0145] Set the perspective projection algorithm and project P c,i onto the image plane, specifically:

[0146] Among them, u i and v i are the pixel coordinates of the i-th point in the image; Z c,i is the depth of the point P c,i , satisfying Z c,i ≠0;

[0147] Given the calibration corresponding point pairs {(P l,i , p i )|i = 1, 2,..., N c};

[0148] Among them, p i = (u i , v i ) T , is the corresponding pixel coordinate in the image; N c is the number of calibration point pairs;

[0149] Set the projection error function as:

[0150] Among them, the projection function Among them, P = (P x , P y , P z ) T ;

[0151] To ensure that R is orthogonal, it is represented by Euler angles:

[0152] R = R x (α)R y(β)R z (γ); where α is the rotation angle about the x-axis, β is the rotation angle about the y-axis, and γ is the rotation angle about the z-axis; R x (α), R y (β) and R z (γ) are the matrices for rotation about the x-axis, y-axis, and z-axis respectively, specifically:

[0153]

[0154] The parameter vector is denoted as θ = (α, β, γ, T x , T y , T z ) T ;

[0155] The parameters are updated using the gradient descent method as follows:

[0156]

[0157] where k is the number of iterations; η > 0 is the step size, whose value is determined by the system sampling period and convergence analysis and can be obtained experimentally during the calibration phase; is the gradient of the error function E with respect to θ, and its j-th component is:

[0158] where θ j is the j-th element of the vector θ.

[0159] By setting up the algorithm for converting the LiDAR coordinates to the camera coordinate system and using the perspective projection algorithm to convert the LiDAR point cloud data to the image plane, the problem of inconsistent coordinate systems between the camera and the LiDAR data is solved, thus achieving high-precision fusion of multi-sensor data. Specifically, first, by setting the rotation matrix and translation vector, the point cloud data collected by the LiDAR is converted to the camera coordinate system to obtain P c,i = R · P l,i+T. Subsequently, the transformed point cloud data is mapped to the image plane through the perspective projection formula. Next, given the calibrated corresponding point pairs and constructing the projection error function, the parameters are iteratively updated by the gradient descent method to ensure that the calibration result makes the lidar data and the image data accurately aligned in the same coordinate system. This series of steps solves the problems of coordinate system differences and data fusion difficulties caused by different sensor data, effectively maps the three-dimensional point cloud data into a two-dimensional image, and provides an accurate geometric basis for subsequent environment recognition and path planning. In addition, using Euler angles to represent the rotation matrix and optimizing the external parameters reduces the calibration error and improves the calibration accuracy and system robustness. Finally, this method realizes the synchronization of multi-sensor data, ensures that the robot can accurately identify the grassland area and obstacles in a complex environment, and then plans a reasonable path. This not only improves the operation efficiency and navigation accuracy of the lawn mowing robot, but also provides a high-precision and low-error solution for similar multi-sensor fusion applications.

[0160] Including grassland area segmentation, specifically:

[0161] For each pixel I(u, v) in the image, whose color components R(u, v), G(u, v), B(u, v) all belong to [0, 255], set the judgment function:

[0162]

[0163] where Γ outputs a binary value, 1 represents the grassland area, and 0 represents the non-grassland area; where Γ outputting a binary value means that Γ output has only two possible values, 0 or 1, to distinguish the grassland area and the non-grassland area;

[0164] Obtain the grassland mask: M g (u, v) = Γ(R(u, v), G(u, v), B(u, v)).

[0165] By setting up a color judgment function for grassland area segmentation, the problem that it is difficult to accurately distinguish grassland from non-grassland areas in complex environments in traditional vision algorithms is solved. Specifically, in this step, the color components of each pixel in the image, namely the red, green, and blue values, are detected, and using the preset threshold judgment conditions, each pixel is classified as: grassland with an output value of 1 or non-grassland with an output value of 0. This binary processing enables the distribution of grassland in the image to be clearly extracted, forming a grassland mask. In this way, through the clear extraction of color features, the system can stably identify grassland areas under various lighting and background conditions, thus avoiding recognition errors caused by color interference or cluttered backgrounds. Specifically, through this step, firstly, the problem of inaccurate recognition caused by factors such as uneven lighting, color changes, or mixed backgrounds is solved; secondly, the binary grassland mask provides accurate area information for subsequent path planning, obstacle detection, and robot navigation, ensuring that the robot can focus on the grassland area during operation and will not stray into non-grassland areas; finally, this method simplifies the data processing flow, quickly generates a grassland mask through simple pixel-level judgment, thereby improving the overall response speed and real-time performance of the system. In summary, by using the color judgment function to segment each pixel in the image and accurately distinguish the grassland area from the non-grassland area, this step not only overcomes the problem of improper handling of complex backgrounds in traditional methods, but also provides a clear and accurate environmental map for subsequent tasks of intelligent lawn mowing robots, thus realizing efficient and accurate path planning and operation execution.

[0166] It also includes obstacle detection, specifically:

[0167] Using lidar point cloud data, calculate the 10th percentile for all Z l,i values to obtain the ground height:

[0168]

[0169] Among them, Percentile(*, 10%) represents the 10th percentile in the set, reflecting the typical ground height;

[0170] For each point P l,i =(X l,i , Y l,i , Z l,i ), set the height difference: h i =Z l,i -Z ground ;

[0171] Among them, h i is the height difference of the i-th point relative to the ground;

[0172] Set the obstacle height threshold, denoted as h th, determined by the height difference between the ground and non-ground objects, set a judgment function:

[0173]

[0174] Among them, Δ(h i ) = 1 indicates that this point is an obstacle;

[0175] Generate an obstacle mask: M o (X l,i , Y l,i ) = Δ(h i ).

[0176] Obstacle detection is carried out by calculating the ground height and setting a height difference threshold, which solves the problem of obstacle recognition errors caused by terrain undulation and noise in traditional methods. Specifically, this step first calculates the 10th percentile of all points using the lidar point cloud data as the ground height reference to avoid ground estimation deviation caused by extreme high or low values. Then, by calculating the height difference of each point and comparing it with the preset obstacle height threshold, it accurately determines whether this point belongs to an obstacle. Finally, all the points determined to be obstacles together form an obstacle mask, providing reliable data for subsequent path planning and obstacle avoidance decision-making. Through the above steps, the problem of misjudgment caused by terrain changes is first solved. Traditional methods may be affected by local high and low undulations, resulting in normal ground being misidentified as an obstacle or low obstacles being ignored. However, through the percentile statistical method, the interference of abnormal points can be reduced while ensuring accuracy, making the ground height calculation more stable. Secondly, this method effectively meets the detection requirements of objects at different heights and can adaptively adjust the obstacle detection standard. For example, in rough terrain, the ground height will fluctuate, but since this method is based on statistical analysis, it will not easily misjudge normal terrain changes as obstacles, improving the robustness of detection. In addition, the generated obstacle mask can provide clear environmental perception data for intelligent devices, enabling them to accurately identify obstacles ahead and adjust the path in advance, thereby improving the obstacle avoidance ability and ensuring operation safety. Compared with traditional methods, this method is computationally efficient, can update obstacle information in real time, and is applicable to complex dynamic environments. Generally speaking, this step determines the ground height through statistical methods, combines height difference calculations to detect obstacles, improves detection accuracy, reduces misjudgment, and provides accurate data support for subsequent path optimization and obstacle avoidance strategies.

[0177] It also includes path planning, specifically:

[0178] Record the grassland area obtained by back-projecting the image as Back-projection is an existing technology in computer vision;

[0179] Among them, X, Y are ground coordinates; Indicates that this position is a grassland area, and 0 indicates a non-grassland area;

[0180] Meanwhile, the obstacles detected by the lidar are mapped to obtain an obstacle mask, denoted as M o (X, Y);

[0181] Among them, M o (X, Y) = 1 indicates that there is an obstacle at this position, and 0 indicates no obstacle;

[0182] Set the feasible mowing area as: Indicates that only the area that is both a grassland area and has no obstacles is passable;

[0183] Discretize the feasible area into grids with a fixed grid resolution r (unit: meter, determined by the size of the robot, such as 1 / 2 of the robot's width). Let the grid set be:

[0184]

[0185] Among them, X p,q , Y p,q Are the coordinates of the center of the (p, q)-th grid respectively; N x , N y Are the number of grids in the horizontal and vertical directions respectively;

[0186] Let the starting point of the mowing robot be P s =(X s , Y s ) and the target end point be P e =(X e , Y e );

[0187] Obtain the straight-line distance between the starting point and the end point, denoted as L0;

[0188] Let the distance between the midpoints of the expected path be d s ;

[0189] Then the number of initial path points N is: Is rounded up;

[0190] Set the linear interpolation algorithm and generate the initial path, specifically:

[0191]

[0192] Among them, Is the starting point, Is the end point, and the remaining points are evenly distributed interpolation points.

[0193] Through grassland area back-projection, obstacle mapping, feasible area calculation, and grid path generation, accurate path planning is achieved, solving the problems of poor adaptability to complex environments, ineffective obstacle avoidance, and inaccurate movement of traditional path planning for lawn mowing robots. Specifically, first, computer vision technology is used to back-project the camera image, converting the grassland area to the ground coordinate system to ensure accurate differentiation between feasible and non-feasible areas during path planning. Second, the obstacle data detected by lidar is mapped to the same coordinate system, and an obstacle mask is generated to exclude non-passable areas, ensuring that the robot's path does not overlap with obstacles. Then, by cross-calculating the grassland area and the obstacle mask, the final feasible mowing area is determined, allowing passage only in grassland areas without obstacles, thereby improving the accuracy of path planning. Next, the feasible area is grid-processed, discretizing the ground in units of half the robot's size, making the path points more regular and facilitating the planning of a smooth path that conforms to the robot's motion characteristics. During path generation, first, the straight-line distance between the starting point and the target point is calculated, and the path point interval is set. The linear interpolation method is used to generate the initial path, making the path points evenly distributed to ensure that the robot moves along a reasonable trajectory. Compared with traditional methods, this solution can update the path in real time based on vision and lidar information to adapt to dynamic environmental changes. At the same time, the grid method makes path generation more systematic and reasonable, avoiding unnecessary twists or detours in the path and improving the navigation efficiency of the robot in complex terrains. Finally, this method ensures that the lawn mowing robot can complete the work efficiently and safely, reducing ineffective travel, optimizing the mowing coverage rate, and improving the overall efficiency of the mowing operation.

[0194] Refer to Figure 1 , and it also includes obstacle safety constraint adjustment, specifically:

[0195] S32. Obstacle safety constraint adjustment:

[0196] For each intermediate point P i =(X i , Y i ), the following steps are executed:

[0197] S321. Calculate the nearest obstacle distance:

[0198] For point P i , calculate its Euclidean distance to all obstacle grids and take the minimum value:

[0199]

[0200] Where are all obstacle grids that satisfy M o (X, Y)=1; X G , Y Gis the central coordinate of the obstacle grid G;

[0201] S322. Determine safety and calculate the adjustment vector:

[0202] Let the safety distance be d min ;

[0203] If D obs (P i ) < d min , then adjustment is required. The specific adjustment is as follows:

[0204] Let G * be the obstacle grid that minimizes the distance, that is:

[0205]

[0206] Set the unit vector pointing from the obstacle to P i as:

[0207]

[0208] Calculate the adjustment vector:

[0209] Update the point position: P i ← P i + ΔP i ;

[0210] S323. Iterative adjustment:

[0211] Repeat steps S321 and S322 until the corresponding point P i satisfies D obs (P i ) ≥ d min .

[0212] By calculating the distance to the nearest obstacle, judging safety, adjusting waypoints, and iteratively optimizing the path, the path is made to conform to safety constraints, solving the problems of the path being too close to obstacles and insufficient driving safety that may occur in traditional path planning, and ensuring that the lawn mowing robot can operate safely and efficiently in complex environments. First, in step S321, the Euclidean distance from each intermediate path point to all obstacle grids is calculated, and the minimum value is taken to measure the shortest distance between the path point and the obstacle. This calculation ensures that subsequent adjustments can be optimized for the nearest obstacle, preventing the robot from entering high-risk areas. Next, in step S322, a safety distance threshold is set, and it is judged whether the path point meets the safety requirements. If the distance from the path point to the obstacle is less than the safety distance, adjustment is required. To this end, first determine the obstacle grid closest to the path point, and calculate the unit vector pointing from the obstacle to the path point to ensure that the adjustment direction is away from the obstacle. Subsequently, calculate the adjustment vector according to this unit vector, and update the new position of the path point to offset it towards the safe area, thereby reducing the collision risk. In step S323, the path is iteratively optimized to ensure that all path points meet the safety distance requirements. If a certain point still does not meet the safety constraints, repeat the above steps and continuously adjust its position until all path points are in the safe area. This iterative process ensures the optimization of the path in terms of safety and rationality, avoiding excessive deviation of the path caused by insufficient or excessive adjustment at one time. Compared with traditional methods, this solution optimizes the path point by point, ensuring that each path point is within the safe range, rather than simply globally offsetting the path, avoiding problems such as unnatural paths or excessive detours. In addition, through Euclidean distance calculation and unit vector adjustment, the path optimization is more refined, improving the robot's navigation ability in complex environments, ensuring that the path can effectively avoid obstacles and maintain the shortest possible driving distance, thereby improving operation efficiency and reducing energy consumption. Finally, this method can significantly improve the safety and stability of the lawn mowing robot, enabling it to smoothly execute tasks in an environment with obstacles.

[0213] It also includes local center tracking adjustment, specifically:

[0214] S33. Local center tracking adjustment:

[0215] For each intermediate point P i =(X i , Y i ), the following steps are executed:

[0216] S331. Determine the local neighborhood:

[0217] Around the point P i using the point P iConstruct a square neighborhood centered at a point with side length \(l\) (unit: meter), which is determined by the robot size and regional resolution. For example, \(l = 0.5m\). Denote the set of grids contained in this neighborhood as:

[0218]

[0219] S332. Calculate the local centroid:

[0220] Calculate the average value of the center points of all feasible grids in the neighborhood:

[0221]

[0222] where is the number of grids in the set ; \(X\) G , \(Y\) G are the coordinates of the center of each grid;

[0223] S333. Calculate the deviation and gradient:

[0224] Set the deviation distance as:

[0225] Calculate the gradient of this distance with respect to \(P\) i , and the direction is from \(P\) c,i to \(P\) i :

[0226]

[0227] S334. Update the point position:

[0228] Update the point position:

[0229] Repeat steps S331 to S334 until the displacement between two adjacent updates is less than \(0.005m\), and after the update, it still satisfies \(D\) obs (\(P\) i ) \(\geq d\) min . If \(D\) obs (\(P\) i ) \(\geq d\) min is not satisfied, then first return to step S32 for safety adjustment.

[0230] By constructing a local neighborhood, calculating the local centroid, determining the adjustment direction, and iteratively optimizing the path points, the path is made smoother, solving the problem of path deviation or sudden curvature change caused by obstacle avoidance adjustment, ensuring that the mowing robot's driving path is as centered as possible in the feasible area on the premise of safety, and improving the continuity of the path and the operation efficiency. First, in step S331, for each intermediate path point, a square local neighborhood centered at this point is constructed, and its side length is determined by the robot size and the regional resolution, ensuring a reasonable neighborhood range that can contain enough feasible area points for calculation without being too large to cause unstable optimization. By defining the neighborhood grid set, it is ensured that subsequent calculations are only based on the feasible area, improving the effectiveness of the adjustment. Next, in step S332, the average value of the center points of all feasible grids within this neighborhood, that is, the local centroid, is calculated as the ideal driving center position. This step ensures that the calculated centroid will not deviate due to a single abnormal point by averaging all feasible area points, and at the same time makes the path points fit as closely as possible to the center of the feasible area, improving the rationality of the path. Then, in step S333, the deviation of the path point relative to the local centroid is calculated, and the direction of the deviation gradient is used to guide the adjustment direction. By calculating the unit vector from the current path point to the local centroid, it is ensured that the path adjustment direction always points to the optimal center of the feasible area instead of blindly deviating, improving the accuracy of the adjustment. In step S334, the path points are updated according to the calculated adjustment direction and iteratively optimized until the displacement between two adjacent adjustments is less than 0.005 m, ensuring the smooth convergence of the path. If the updated path points exceed the feasible area, return to step S32 for safety adjustment to ensure that the path not only meets local smoothness but also does not enter the obstacle area due to adjustment. Compared with the traditional method, this solution makes the path optimization more refined through local neighborhood calculation, avoids unnecessary detours caused by obstacle avoidance adjustment, and improves the smoothness of the driving path and the operation efficiency. At the same time, by using the gradient optimization method, the path is gradually adjusted to the local optimum instead of making a one-time jump adjustment, reducing the discontinuity of the path and making the robot drive more smoothly. In addition, this method combines safety constraint adjustment and local center tracking adjustment to ensure that the path can avoid obstacles and at the same time keep as close as possible to the center of the feasible area, improving the robot's autonomous navigation ability in complex environments, ultimately improving the operation efficiency and reducing energy consumption.

[0231] It also includes path smoothness optimization, specifically:

[0232] S34. Path smoothness optimization:

[0233] For three consecutive points P i-1 =(X i-1 , Y i-1 ), P i =(X i , Y i ) and P i+1=(X i+1 ,Y i+1 ), perform the following steps:

[0234] S341. Calculate the included angle:

[0235] Calculate the vector:

[0236]

[0237] Then the included angle calculation is:

[0238]

[0239] Where is the Euclidean norm of the vector ; is the Euclidean norm of the vector ;

[0240] S342. Smoothness judgment and update:

[0241] Obtain the minimum turning radius of the lawn mowing robot, denoted as R min ;

[0242] Let the maximum allowable included angle be φ max ,

[0243] where Δs is the average distance between adjacent points in the path;

[0244] If there exist three consecutive points that satisfy: φ(P i-1 ,P i ,P i+1 )>φ max , then directly smooth and update the middle point: This update can cause the turning angle of the path to drop suddenly, ensuring smoothness. After the update, it is necessary to check whether the updated point still satisfies the obstacle safety and local center conditions. If not, return to the previous stage in turn for readjustment;

[0245] S343. Iterative smoothing:

[0246] Execute steps S341 to S342 for all consecutive three points in the path in turn until all φ(P i-1 ,P i ,P i+1 ) in the entire path are not greater than φ max .

[0247] By calculating the angles between path points, determining whether they exceed the smoothness threshold, and smoothly adjusting the points that exceed the threshold, the curvature change of the path becomes smoother, solving the sharp turn problem caused by uneven path point distribution or obstacle avoidance adjustment, and improving the feasibility of the path and the stability of the robot's movement. First, in step S341, for three consecutive points in the path, two adjacent vectors are calculated, and the angle is calculated using the vector angle formula to ensure that the local curvature of the path is measurable. This calculation method is based on the Euclidean distance, making the angle calculation stable and reliable, capable of accurately reflecting the degree of path turning, and providing data support for subsequent smoothing optimization. Then, in step S342, first obtain the minimum turning radius of the robot and calculate the maximum allowable angle to ensure that the path is within the robot's motion ability range, enabling the robot to smoothly pass through the path with the minimum turning radius. If it is found that the angle of a certain section of the path exceeds the allowable value, directly adjust the position of the middle point so that it falls on the line connecting the two end points to reduce the turning angle. After adjustment, it is necessary to check whether the updated point still meets the obstacle safety conditions and local center tracking requirements to ensure that the optimization does not cause new safety hazards or lead to the path deviating from the optimal feasible area. Then, in step S343, traverse all consecutive three points, repeat the angle calculation and smoothing adjustment until the angles of all path points meet the constraint of the maximum allowable angle, thereby achieving global path smoothing optimization. This iterative method ensures that there are no sharp turns in the path, and the optimized path is smoother as a whole, improving the robot's autonomous driving ability in complex environments. Compared with traditional path optimization methods, this scheme realizes path smoothness control through the angle analysis and optimization of local path points, enabling the robot to smoothly drive along the path and avoiding driving difficulties or out-of-control caused by sharp turns. At the same time, due to the combination of the robot's own kinematic limit of the minimum turning radius in the optimization process, the feasibility of the path is ensured. In addition, this method can be dynamically adjusted and return to the previous obstacle safety constraint or local center tracking adjustment step when necessary to ensure that the final path meets both the smoothness requirements and does not affect safety and path rationality, improving the stability and working efficiency of the robot in actual operations.

[0248] It should be noted that in this article, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or device comprising a series of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article or device.

[0249] The above are only the preferred embodiments of the present invention. It should be noted that for those of ordinary skill in the art, without departing from the technical principle of the present invention, several improvements and refinements can be made, and these improvements and refinements should also be regarded as the protection scope of the present invention.

Claims

1. An intelligent robot path planning method based on visual recognition, characterized in that, It includes: Record the image captured by the camera as I(u, v); Where u is the horizontal pixel coordinate of the image, and its value range is [0, W - 1]; v is the vertical pixel coordinate of the image, and its value range is [0, H - 1]; W is the width of the image; H is the height of the image; Let the camera internal parameter matrix be K, specifically: where f x and f y are the focal lengths in the horizontal and vertical directions respectively; c x and c y are the horizontal and vertical coordinates of the principal point of the image respectively; Let the lidar collect the point cloud set as Among them, P l,i is the three-dimensional coordinate of the i-th point; X l,i , Y l,i and Z l,i are the x, y, and z coordinates of the i-th point in the lidar coordinate system respectively; N l is the total number of point clouds.

2. The method for path planning of an intelligent robot based on visual recognition according to claim 1, wherein It also includes data synchronization and camera-lidar calibration, specifically: Set the lidar-to-camera coordinate system conversion algorithm and convert lidar points to the camera coordinate system, specifically: Let P l,i =(X l,i , Y l,i , Z l,i ) T ; Then convert to the camera coordinate system to obtain: P c,i = R·P l,i + T, where P c,i = (X c,i , Y c,i , Z c,i ) T ; Among them, is a rotation matrix, and its elements are rotation coefficients; T = (T x , T y , T z ), T is a translation vector, and the components T x , T y and T z represent the translation amounts along the x, y, and z directions respectively; Among them, is the set of real numbers; the rotation matrix R satisfies R T R = I, R T is the transpose of R, I is a 3×3 identity matrix, and at the same time satisfies det(R) = 1; Set up the perspective projection algorithm and project P c,i onto the image plane, specifically as follows: where u i and v i are the pixel coordinates of the i-th point in the image; Z c,i is the depth of point P c,i . Given calibration corresponding point pairs \(\{(P l,i , p i )|i = 1, 2, \ldots, N c \}; where p i = (u i , v i ) T is the corresponding pixel coordinate in the image; N c is the number of calibration point pairs; Set the projection error function as follows: Among them, the projection function where P = (P x , P y , P z ) T ; To ensure the orthogonality of R, it is represented by Euler angles: R = R x (α)R y (β)R z (γ); where α is the rotation angle about the x-axis, β is the rotation angle about the y-axis, and γ is the rotation angle about the z-axis; R x (α), R y (β) and R z (γ) are the matrices for rotation about the x-axis, y-axis, and z-axis respectively, specifically: The parameter vector is denoted as θ = (α, β, γ, T x , T y , T z ). T ; Use the gradient descent method to update the parameters, specifically as follows: where \(k\) is the number of iterations; \(\eta>0\) is the step size; is the gradient of the error function \(E\) with respect to \(\theta\), and its \(j\)-th component is: where θ j is the j-th element of the vector θ.

3. An intelligent robot path planning method based on visual recognition according to claim 2, characterized in that, It includes grassland area segmentation, specifically: For each pixel I(u, v) in the image, its color components R(u, v), G(u, v), and B(u, v) all belong to [0, 255]. Set the judgment function: Where Γ outputs a binary value, 1 represents the grassland area, and 0 represents the non-grassland area; Obtain the grassland mask: M g (u, v) = Γ(R(u, v), G(u, v), B(u, v)).

4. An intelligent robot path planning method based on visual recognition according to claim 3, characterized in that, It also includes obstacle detection, specifically: Using lidar point cloud data, calculate the 10th percentile for all Z l,i values to obtain the ground height: Where Percentile(*, 10%) represents the 10th percentile in the set; For each point P l,i =(X l,i , Y l,i , Z l,i ), set the height difference: h i =Z l,i -Z ground ; where h i is the height difference of the i-th point relative to the ground; Set an obstacle height threshold, denoted as h th , and set a decision function: where Δ(h i ) = 1 indicates that this point is an obstacle; Generate obstacle mask: M o (X l,i , Y l,i ) = Δ(h i ).

5. A method for path planning of an intelligent robot based on visual recognition according to claim 4, characterized in that, It also includes path planning, specifically: The grassland area obtained by back-projecting the image is denoted as Among them, X and Y are ground coordinates; Indicates that this position is a grassland area, and 0 indicates a non-grassland area; Meanwhile, the obstacle detected by the lidar is mapped to obtain an obstacle mask, denoted as M o (X,Y); Among them, M o (X, Y) = 1 indicates that there is an obstacle at this position, and 0 indicates no obstacle; Set the feasible mowing area as: Discretize the feasible region into a grid with a fixed grid resolution r. Let the grid set be: Among them, X p,q , Y p,q are the coordinates of the center of the (p, q)-th grid respectively; N x , N y are the numbers of grids in the horizontal and vertical directions respectively. Let the starting point of the lawn mowing robot be P s =(X s ,Y s ) and the target end point be P e =(X e ,Y e ); Obtain the straight-line distance between the starting point and the ending point, denoted as L0; Let the distance between the midpoints of the expected path be d s ; Then the initial number of path points N is: It is rounded up; Set the straight-line interpolation algorithm and generate the initial path, specifically: Among them, as the starting point, as the ending point, and the remaining points are evenly distributed interpolation points.

6. The intelligent robot path planning method based on visual recognition according to claim 5, characterized in that, It also includes obstacle safety constraint adjustment, specifically: S32. Obstacle safety constraint adjustment: For each intermediate point P i =(X i , Y i ), the following steps are performed: S321. Calculate the distance to the nearest obstacle: For point P i , calculate its Euclidean distance to all obstacle grids and take the minimum value: Among them, is the obstacle grid that satisfies M o (X,Y) = 1; X G , Y G is the center coordinate of the obstacle grid G; S322. Judge safety and calculate the adjustment vector: Set the safety distance as d min ; If D obs (P i )<d min , adjustment is required. The specific adjustment is as follows: Let G * be the obstacle grid that minimizes the distance, i.e.: Set a unit vector pointing from the obstacle to P i as follows: Calculate the adjustment vector: Updated point position: P i ←P i +ΔP i ; S323. Iterative adjustment: Repeat steps S321 and S322 until the corresponding point P i satisfies D obs (P i ) ≥ d min .

7. An intelligent robot path planning method based on visual recognition according to claim 6, characterized in that, It also includes local center tracking adjustment, specifically: S33. Local center tracking adjustment: For each intermediate point P i =(X i , Y i ), perform the following steps: S331. Determine the local neighborhood: At point P i Construct a square neighborhood centered at point P i with side length l. Denote the set of grids contained within this neighborhood as: S332. Calculate the local centroid: Calculate the average value of the central points of all feasible grids within the neighborhood: Among them, is the number of grids in the set ; X G , Y G are the coordinates of the center of each grid; S333. Calculate the deviation and gradient: Set the deviation distance to: Compute the gradient of this distance with respect to P i in the direction from P c,i towards P i : S334. Update the point position: Update point location: Repeat steps S331 to S334 until the displacement between two adjacent updates is less than 0.005 m and D obs (P i )≥d min is still satisfied after the update. If D obs (P i )≥d min is not satisfied, return to step S32 first for safety adjustment.

8. An intelligent robot path planning method based on visual recognition according to claim 7, characterized in that, It also includes path smoothness optimization, specifically: S34. Path smoothness optimization: For three consecutive points P i-1 =(X i-1 ,Y i-1 ), P i =(X i ,Y i ), and P i+1 =(X i+1 ,Y i+1 ), perform the following steps: S341. Calculate the included angle: Calculate the vector: Then the included angle calculation is: Among them, is the Euclidean norm of the vector ; is the Euclidean norm of the vector ; S342. Smooth judgment and update: Obtain the minimum turning radius of the lawn mowing robot, denoted as R min ; Let the maximum allowable included angle be φ max , Where Δs is the average distance between adjacent points in the path; If there exist three consecutive points that satisfy: φ(P i-1 ,P i ,P i+1 )>φ max , then directly smooth and update the middle point: S343. Iterative smoothing: Steps S341 to S342 are sequentially executed for all consecutive three points in the path until all φ(P i-1 ,P i ,P i+1 ) are not greater than φ max .

Images