Graph-free full-coverage operation control method and equipment for intelligent mowing robot and medium

Abstract

The invention discloses a graph-free full-coverage operation control method and device for an intelligent mowing robot and a medium, and the method comprises the steps: controlling a controlled intelligent mowing robot to start to run from a starting point position, and enabling the controlled intelligent mowing robot to start to run in the running process of the controlled intelligent mowing robot; acquiring an image of the controlled intelligent mowing robot in the advancing direction acquired by a visual sensor; segmenting the currently acquired image, and identifying an obstacle; marking a decision area in the currently acquired image; the position relation between the obstacle area and the decision area is judged, and a current action decision is determined according to the judgment result; and acquiring a yaw angle output by fusion of the vision and the IMU, calculating a compensation parameter according to the currently obtained action decision and the currently obtained yaw angle, and sending the compensation parameter to a control end of the controlled intelligent mowing robot until a completely covered mowing task is executed. According to the method, full-coverage operation control can be efficiently realized without depending on a priori map and a coverage path.

Description

Technical Field

[0001] The present invention relates to the technical field of intelligent lawn mowers, and in particular to a method, equipment and medium for controlling the non-mapped full-coverage operation of an intelligent lawn mowing robot. Background Art

[0002] Traditional manual mowing requires a lot of time and physical effort, while smart mowing robots can complete the task automatically, greatly reducing manpower input. In the existing technology, smart mowing robots usually complete full-coverage mowing with the help of global positioning information provided by global positioning systems such as GPS / RTK and real-time path tracking. First, the global positioning information is used to provide boundary information of the mowing area, and the working area of ​​the mowing robot is determined, that is, the prior work area map. Then, the coverage path is planned according to the working area. Finally, the robot performs motion control based on the current positioning information and the globally planned coverage path, and controls the mowing robot to track the path points to complete the full-coverage mowing task. In order to avoid the problem of positioning loss caused by local occlusion, the existing technology usually uses visual SLAM (simultaneous localization and mapping or concurrent mapping and positioning) or laser SLAM to provide more reliable global positioning on the basis of the global positioning system to complete the full coverage and control of the mowing area.

[0003] However, a single GPS / RTK global positioning system may have local occlusions that cause unreliable positioning. Although the introduction of visual SLAM or laser SLAM can effectively solve the problem of inaccurate local positioning, the use of multiple sensors and the construction of a global boundary map require high hardware costs, which are not suitable for miniaturized low-cost mowing robots. In addition, since it relies on the use of prior maps to generate coverage paths, it is necessary to build a cumbersome boundary map, and the implementation process is relatively complicated. Especially when facing large mowing areas, the construction of the prior working area (boundary) map is very time-consuming, resulting in low real-time mowing efficiency. Summary of the Invention

[0004] The technical problem to be solved by the present invention is: in response to the technical problems existing in the prior art, the present invention provides a method, equipment and medium for controlling the full coverage operation of an intelligent lawn mowing robot without a map, which is simple to implement, low in cost, and high in control efficiency and reliability. It can efficiently achieve full coverage operation control without relying on prior maps and coverage paths.

[0005] In order to solve the above technical problems, the technical solution proposed by the present invention is:

[0006] A method for controlling the full-coverage operation of an intelligent lawn mowing robot without a map is applied to the intelligent lawn mowing robot. The intelligent lawn mowing robot is equipped with at least one visual sensor and at least one IMU. The visual sensor is arranged at the forward direction end of the intelligent lawn mowing robot. The method comprises the following steps:

[0007] Controlling the controlled intelligent lawn mower robot to start moving from a starting position, and obtaining an image of the controlled intelligent lawn mower robot's moving direction collected by a visual sensor during the moving process of the controlled intelligent lawn mower robot;

[0008] The currently acquired image is segmented into grass and non-grass areas using a segmentation model, and obstacles are identified, grass and non-grass areas are segmented, and obstacle areas are marked;

[0009] Determining a decision area in the currently acquired image according to the width of the controlled intelligent lawn mowing robot and the segmented grass area, wherein the decision area corresponds to the forward movement area of ​​the controlled intelligent lawn mowing robot in the image;

[0010] Determine the positional relationship between the obstacle area and the decision area, and determine a current action decision based on the determination result, wherein the action decision includes going straight, turning around, avoiding obstacles, and stopping;

[0011] Obtain the yaw angle output by the fusion of vision and IMU, calculate the compensation parameter based on the currently obtained action decision and the currently obtained yaw angle, and send it to the control end of the controlled intelligent mowing robot to control the controlled intelligent mowing robot to execute the current action decision until the full coverage mowing task is performed.

[0012] Furthermore, the left and right boundary lines of the decision area correspond to the left and right reference trajectory lines of the controlled intelligent lawn mower robot in the forward direction, respectively. The reference trajectory lines are extended for a specified length from the end close to the controlled intelligent lawn mower robot in the image as the starting point toward the forward direction of the controlled intelligent lawn mower robot, and the distance between the two reference trajectory lines is greater than the width of the controlled intelligent lawn mower robot.

[0013] Furthermore, it also includes setting a decision line parallel to the bottom of the image according to the maximum steering angle of the controlled intelligent lawn mowing robot, dividing the area above the decision line in the decision area into a main decision area Area_w, and the height of the decision line from the bottom of the image is configured so that the controlled intelligent lawn mowing robot does not touch the obstacle in front when moving at the maximum steering angle. Whether to perform a straight-line action is determined based on whether there is an obstacle in the main decision area Area_w, and the left area Area_left and the right area Area_right are used as stop auxiliary judgment areas to assist in determining whether to perform a U-turn or obstacle avoidance action. The left area Area_left is the area between the left reference trajectory line and the left boundary of the image, and the right area Area_right is the area between the right reference trajectory line and the right boundary of the image.

[0014] Furthermore, the determining of the positional relationship between the obstacle area and the decision area and determining the current action decision according to the determination result includes:

[0015] Determine whether there is an obstacle in the main decision area Area_w. If yes, proceed to determine whether there is an obstacle in the left area Area_left and the right area Area_right. If not, perform a straight-ahead action.

[0016] Determine whether there are obstacles in the left area Area_left and the right area Area_right. If it is determined that there are no obstacles or there is an obstacle in the right area Area_left, perform the left obstacle avoidance action. If it is determined that there is an obstacle in the left area Area_right, perform the right obstacle avoidance action. If it is determined that there are obstacles in both the left area Area_left and the right area Area_right, perform a U-turn.

[0017] Furthermore, it also includes using the t1 area and t2 area located below the decision line in the left area Area_left and the right area Area_right as auxiliary stop judgment areas to assist in determining whether to perform a stop action. When it is determined that the area of ​​the unknown area in the main decision area Area_w exceeds a preset threshold, the area of ​​the unknown area in the t1 area and the t2 area is determined. If the area of ​​the unknown area in the t1 area or the t2 area exceeds the preset threshold, the stop action is performed.

[0018] Furthermore, when performing an obstacle avoidance action, the area occupied by the obstacle pixels is obtained, and the steering angle corresponding to the current area occupied by the obstacle pixels is queried from the steering angle database. The controlled intelligent lawn mowing robot is controlled to turn according to the steering angle obtained in the query to complete the obstacle avoidance. The steering angle database stores the required steering angles corresponding to different obstacle pixel occupied areas.

[0019] Furthermore, when executing a U-turn, the starting position information of the controlled intelligent mowing robot before the U-turn is recorded, and the robot turns at a specified turning angle from the current position. If the real-time position information of the controlled intelligent mowing robot meets the starting position information, d =|D-(x c -x a )|≤T d , then stop the movement, and then adjust the robot movement with the maximum steering mode and yaw reference benchmark. If the yaw angle e<T e When the conditions are met, the movement stops and the U-turn task is completed, where D represents the full coverage row spacing, x c Indicates the horizontal offset of the position point after the real-time U-turn is completed, x aIndicates the lateral offset of the position point when starting the U-turn, T d represents the deviation threshold, T e Indicates the yaw deviation threshold.

[0020] Furthermore, one or more visual sensors are installed on the front and rear of the controlled intelligent lawn mowing robot respectively. When the controlled intelligent lawn mowing robot is traveling along the forward path, the image captured by the front visual sensor is obtained to determine the current action decision. After completing the mowing task of the forward path, when it needs to return to perform the mowing task of the backward path, the controlled intelligent lawn mowing robot is controlled to travel in a reverse motion mode, and when the controlled intelligent lawn mowing robot is traveling backward, the image captured by the rear visual sensor is obtained to determine the current action decision.

[0021] An intelligent lawn mowing robot device comprises a processor and a memory, wherein the memory is used to store computer programs, and the processor is used to execute the above method.

[0022] A computer-readable storage medium storing a computer program, wherein the computer program implements the above method when executed by a processor.

[0023] Compared with the prior art, the beneficial effect of the present invention is that: the present invention uses a visual sensor to collect images during the driving of the lawn mower, and uses image segmentation and obstacle recognition to segment the grass area and the non-grass area and mark the obstacle area, and then calibrates the decision area in the currently acquired image, and then determines the action decision that needs to be executed according to the positional relationship between the decision area and the obstacle area. Finally, the compensation error is calculated in combination with the yaw angle output by the IMU and sent to the robot control end, and the robot is controlled to complete real-time action decisions until the full coverage mowing task is performed. The visual sensor and the IMU sensor can be used to realize the map-free full coverage operation control of the intelligent lawn mower robot, without relying on any prior map and "point tracking". It can not only avoid the problem of affecting the control accuracy due to inaccurate GPS / RTK positioning data, improve the accuracy and reliability of the real-time control of the intelligent lawn mower robot, but also avoid the complex prior map construction process, greatly reduce the complexity of the control, improve the efficiency and real-time performance of the control, and at the same time greatly reduce the implementation cost. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 This is a schematic diagram of the implementation process of the first embodiment of the present invention to realize the full coverage operation control of the intelligent lawn mowing robot without a map.

[0025] Figure 2 This is a schematic diagram of the principle of the decision area generated in Example 1 of the present invention.

[0026] Figure 3 Schematic diagram of the principle of the Ackerman vehicle model used in Example 1 of the present invention.

[0027] Figure 4 This is a schematic diagram of the implementation flow of the action decision logic in Example 1 of the present invention.

[0028] Figure 5 This is a schematic diagram of the effect of the full coverage path achieved in Example 2 of the present invention. DETAILED DESCRIPTION

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific preferred embodiments, but the scope of protection of the present invention is not limited thereby.

[0030] For ease of understanding, the relevant technical background of the present invention is first introduced by way of example.

[0031] In the prior art, the operation control of intelligent lawn mowing robots is mainly achieved in the following ways:

[0032] 1. Use global positioning systems such as GPS / RTK to provide boundary information of the mowing area. Based on this boundary information, use Boustrophedon ("ox plowing method") or random covering method to generate a global coverage path. The mowing robot performs full coverage motion control based on the generated coverage path and the current real-time global positioning information of the robot.

[0033] 2. Use lidar or vision for SLAM, build a global point cloud map and a global work area boundary map based on the SLAM positioning information, and use the "ox plowing method" or other methods based on the boundary map to plan a full coverage path based on the prior boundary map. Plan the full coverage control of the path points based on the current positioning information of the mowing robot in the map.

[0034] 3. Use multi-sensor fusion to provide more reliable positioning information, and combine it with the above 1 or 2 methods to complete full coverage path planning and control.

[0035] 4. Use vision-based deep learning to segment the mowed grass areas and uncut grass areas in the mowing area, and use the segmented edge information to guide the movement of the mowing robot to achieve full coverage control. This type of method no longer relies on positioning information and path planning.

[0036] For the above-mentioned control methods, first of all, the first method based on GPS / RTK, the second method based on multi-sensor fusion, and the third method using lidar or vision for SLAM all need to rely on positioning information to build a boundary map of the working area of ​​the lawn mowing robot, and then plan a full-coverage motion control path in advance based on the boundary map. The lawn mowing robot then completes the control operation based on the real-time positioning information of the global positioning. During the actual coverage movement, the coverage is completed by "point tracking" based on the current positioning information of the lawn mowing robot and the planned coverage path information. These methods are highly dependent on the GPS / RTK global positioning system, which is prone to inaccurate positioning information due to occlusion of the equipment, which in turn leads to the inability to obtain correct boundary map information or complete failure of point tracking in motion control.

[0037] While the second approach, using radar for SLAM, can avoid the problem of positioning loss due to partial occlusion, it also fails when the lidar cannot detect the surrounding environment in open environments and cannot obtain any positioning information. The same problem applies to SLAM using visual sensors, and when the mowing area is large enough, it often takes longer to build a map and generate a full coverage path. The third approach combines vision / lidar devices with global positioning systems such as GPS / RTK, achieving multi-sensor positioning fusion through algorithms such as VINS-Fusion. While this effectively addresses the global positioning system's inability to complete full coverage path planning and control due to partial occlusion, as well as the problem of vision / lidar sensor failure in large scenes, the high hardware cost of multiple sensors hinders the miniaturization and practical application of intelligent lawn mowing robots. Furthermore, the operation of multiple sensors requires time synchronization in both hardware and software. Higher time synchronization accuracy improves control precision, which makes actual control more difficult and complex.

[0038] The fourth vision-based deep learning method can complete the full coverage operation of the mowing robot in some scenarios by using only visual sensors. It does not need to rely on any positioning information, nor does it need to build any work area boundary map in advance. The hardware cost is low, but a large amount of grassland data set is required for training, especially for the segmentation of mowing areas and unmowing areas. If the grassland has a high proportion of vegetation, it is easy to complete the segmentation, but if the proportion of grassland vegetation is low, it is difficult to perform accurate segmentation, so the actual control accuracy is not high.

[0039] The present invention installs at least one visual sensor and at least one IMU (Inertial Measurement Unit) sensor on the intelligent lawn mowing robot. During the driving process of the intelligent lawn mowing robot, the visual sensor collects images in the direction of the robot's forward movement, segments the images collected by the visual sensor and identifies obstacles, segments the grass area and the non-grass area, and marks the obstacle area. Then, according to the width of the controlled intelligent lawn mowing robot and the segmented grass area, a decision area is calibrated in the currently acquired image. The decision area corresponds to the forward movement area of ​​the controlled intelligent lawn mowing robot in the image. Then, according to the positional relationship between the decision area and the obstacle area, the action decision that needs to be executed is determined. Then, the compensation error is calculated by combining the vision and the yaw angle output by the IMU and sent to the robot control end to control the robot to complete Real-time action decision-making, until the execution of full coverage mowing tasks, can use visual sensors and IMU sensors to simulate the human decision-making process to achieve full coverage operation control of the intelligent mowing robot. The entire process does not need to rely on any prior map and global positioning information, nor does it need to perform global path planning in advance based on the prior map, and no "point tracking" is required. It can not only avoid the problem of control accuracy affected by inaccurate GPS / RTK positioning data, improve the accuracy and reliability of real-time control of the intelligent mowing robot, but also avoid the complex prior map construction process, greatly reduce the complexity of control, improve the efficiency and real-time performance of control, and also greatly reduce the implementation cost, thereby facilitating the miniaturization and wide application of mowing robots.

[0040] In the present invention, the visual sensor is arranged at the forward direction end of the intelligent lawn mowing robot. If the lawn mowing robot cannot achieve on-the-spot turning and can reverse, the forward direction end includes the front end and the rear end of the robot, that is, the visual sensors are installed at the front end and the rear end of the robot. When the robot is traveling forward (positive), the forward direction end is the front end of the robot. When the robot is traveling forward, the visual sensor at the front end is used to collect images. When the robot is traveling backward (positive), the forward direction end is the rear end of the robot. When the robot is traveling backward, the visual sensor at the rear end is used to collect images. When the controlled intelligent lawn mowing robot follows the forward path, During the driving process, the image collected by the front visual sensor is obtained to determine the current action decision. After completing the mowing task of the forward path, when it needs to return to perform the mowing task of the backward path, the controlled intelligent mowing robot is controlled to drive in a reverse motion mode, and during the backward driving of the controlled intelligent mowing robot, the image collected by the rear visual sensor is obtained to determine the current action decision. For the intelligent mowing robot that can realize on-the-spot turning, the forward direction end is the front end of the robot. During the driving process of the controlled intelligent mowing robot, the image collected by the front visual sensor is obtained to determine the current action decision.

[0041] For example, if the mowing robot cannot turn on the spot, a monocular camera can be installed directly in front of and behind the mowing robot to capture images when it is moving forward and backward, respectively. If the mowing robot can stay in place, a visual sensor can be placed directly in front of the smart mowing robot, and a filter can be attached to the camera lens to reduce dizziness caused by strong outdoor light. It is understandable that, of course, more than two visual sensors can be deployed at the forward end of the mowing robot to meet different needs according to actual needs.

[0042] Preferably, the IMU can be arranged to be installed in an upper and lower layout with one of the visual sensors, that is, the IMU is arranged above or below one of the visual sensors, or a camera with an integrated IMU is directly used to avoid the problem of dealing with hardware time synchronization.

[0043] Example 1:

[0044] This embodiment is suitable for a lawn mower robot that cannot achieve on-site steering. A monocular camera is installed in front and behind the lawn mower robot. Figure 1 As shown in FIG, the detailed control steps for realizing the full coverage operation control of the intelligent mowing robot without a map include:

[0045] Step S01: Control the controlled intelligent lawn mower robot to start moving from a starting position. During the moving process of the controlled intelligent lawn mower robot, obtain an image of the moving direction of the controlled intelligent lawn mower robot collected by a visual sensor.

[0046] After the controlled intelligent lawn mower robot is started, the visual sensor is turned on accordingly to collect images. During the driving process of the controlled intelligent lawn mower robot, the visual sensor can collect images in the specified area in the direction of the controlled intelligent lawn mower robot's forward movement in real time. The collected image range is determined by the field of view of the visual sensor.

[0047] Specifically, the visual sensor can be preferably arranged at a central position on the controlled intelligent lawn mowing robot to collect images of the area directly in front of the controlled intelligent lawn mowing robot during its driving process in real time.

[0048] Step S02: Segment the currently acquired image into grass areas and non-grass areas using a segmentation model, identify obstacles, segment the grass areas and non-grass areas, and mark the obstacle areas.

[0049] In this embodiment, various grass data sets collected are preliminarily used to perform segmentation training based on YOLOV8 to obtain the required segmentation model to segment grass and other different targets. After the training is completed, the model obtained can segment grass and non-grass areas.

[0050] It is understandable that the segmentation model can of course also adopt other types of models besides YOLOV8, such as a segmentation model based on a convolutional network CNN or an edge detection algorithm.

[0051] After acquiring real-time images of the controlled intelligent mowing robot's direction of travel from the visual sensor, the captured images are fed into a trained segmentation model to segment grass and non-grass areas. The segmentation results are then used to identify obstacles (such as sprinkler pipes, rocks, and fences), marking the areas in the image where obstacles are present.

[0052] Specifically, after completing the segmentation task using the segmentation model, the segmentation results can also be preprocessed by setting the pixels in the non-grass area of ​​the image to zero, that is, changing the image pixels in the non-grass area to 0, and marking the pixels in the area of ​​typical obstacles (such as sprinkler pipes, etc.) that may appear on the grass (for example, setting the pixels in this area to red). In this way, three situations may appear in one frame of the image, namely, normal grass area, obstacle area and other unknown black area.

[0053] To improve decision robustness, it is preferable to use an additional fast monocular depth estimation function (users can decide whether to enable this function based on the actual sensor type). If using a sensor such as Realsense that can directly obtain image depth information, there is no need to enable the depth estimation function. K-means clustering is performed on the depth image, and the pixels in the corresponding areas determined to be obstacles after segmentation are also marked. When using the monocular depth estimation function, the depth image and RGB image must be aligned.

[0054] Step S03: A decision area is calibrated in the currently acquired image according to the width of the controlled intelligent lawn mowing robot and the segmented grass area. The decision area corresponds to the forward area of ​​the controlled intelligent lawn mowing robot in the image.

[0055] In this embodiment, a decision area is calibrated based on the segmentation results of step S02 to identify a reversing guidance area similar to that in a car's reversing image. This decision area corresponds to the forward travel area of ​​the controlled intelligent lawn mower robot in the image. After the visual sensor is installed, the decision area can be calibrated based on the visual sensor's field of view and the size of the controlled intelligent lawn mower robot. If there are no obstacles within the decision area, the robot can proceed. If there are obstacles within the decision area, it may be due to obstacles in the robot's forward direction that affect its passage, requiring obstacle avoidance control, or it may be due to the robot reaching the limit of its travel and requiring a U-turn. Subsequently, the positional relationship between the decision area and the obstacles can be used to determine the traversable area of ​​the controlled intelligent lawn mower robot, thereby determining the corresponding action decision.

[0056] In this embodiment, the left and right boundaries of the decision area correspond to the left and right reference trajectories of the controlled intelligent lawn mower robot in its forward direction. These reference trajectories extend from the end closest to the controlled intelligent lawn mower robot in the image to a specified length in the direction of the controlled intelligent lawn mower robot's forward movement. The distance between the two reference trajectories is greater than the width of the controlled intelligent lawn mower robot. The left and right boundaries of the decision area are similar to the left and right reference trajectories in a car's reversing image; that is, they correspond to the left and right trajectories of the controlled intelligent lawn mower robot in its forward direction.

[0057] In this embodiment, a decision line parallel to the bottom of the image is set according to the maximum steering angle of the controlled intelligent lawn mowing robot, and the area within the decision area located above the decision line (away from the controlled intelligent lawn mowing robot) is divided into a main decision area Area_w. The height of the decision line from the bottom of the image is configured so that the controlled intelligent lawn mowing robot does not touch the obstacle in front when moving at the maximum steering angle. Whether to execute a straight-line action is determined based on whether there is an obstacle within the main decision area Area_w. The left area Area_left and the right area Area_right are used as auxiliary stop judgment areas to assist in determining whether to execute a U-turn or obstacle avoidance action. The left area Area_left is the area between the left reference trajectory line and the left boundary of the image, and the right area Area_right is the area between the right reference trajectory line and the right boundary of the image.

[0058] by Figure 2 For example, the left and right boundary lines of the decision area are line segments ac and df respectively. Figure 2 The effect shown in the figure is the result of superimposing the left and right boundary lines on the image. Points a and f are located at the bottom of the image. The width of af is approximately the width of the mowing robot itself (to ensure safe passage, the width of af can be slightly larger). be is the decision line. Points b, e, c, and d are all points in the forward direction of the controlled intelligent mowing robot. In the field of view of a real mowing robot, points a, b, and c are collinear. Correspondingly, points d, e, and f are collinear, and line segment abc is parallel to line segment def. In other words, in the field of view of a real mowing robot, ac and df are actually two parallel lines, and the lengths of be and cd are the same as the length of ab. The length of ab (i.e., the height of the decision line from the bottom of the image) is determined by the model of the controlled intelligent mowing robot. For example, if the Ackerman vehicle model is used and the robot's maximum steering angle is 45 degrees, the length of ab must ensure that the robot does not hit obstacles in front of it when moving at the maximum steering angle. The x value is the parameter that needs to be calibrated. Specifically, the length of cb can be set to the length of ab, or can be set to the length of ab / 2, and can be configured according to actual needs.

[0059] In a specific application embodiment, the length of ab needs to ensure that the robot does not touch the obstacle at a distance x in front when moving at the maximum steering angle, where the x value is the parameter that needs to be calibrated. The above x value can be calibrated based on the vehicle's motion and hardware parameters. Taking the Ackerman vehicle model as an example, Figure 3 As shown, for the Ackerman model α is the vehicle's front outer wheel turning angle, β is the vehicle's front inner wheel turning angle, and L is the distance between the vehicle's front and rear axles, which can be obtained based on hardware parameters or directly measured. Assuming the maximum value of θ is 45° and L is 1m, the maximum turning radius can be obtained. This also means that if there is an obstacle 1m in front of the vehicle, then the vehicle can just avoid the obstacle by moving at the maximum steering angle. At this time, R obtains the required x. Furthermore, L can be obtained by adding the safety margin δ based on the obtained x. safe =x+δ, δ is a fine-tuning parameter that can be adjusted according to the actual situation during actual use. safe In fact, it is the value of ab.

[0060] See also Figure 2 After the decision area is calibrated in the image captured by the visual sensor, the image is divided into four large areas: Area_left (o1-o3-ca), Area_w (bcde), Area_right (fd-o4-o6), and Area_n (abef). Area_left and Area_right represent the trapezoidal areas on the left and right sides of ac and df below the dotted line, respectively, which are the areas where the decision area is away from the left and right boundaries of the image. Due to the limited field of view of the camera, the shapes of Area_left and Area_right may be different in practice. The Area_left and Area_right areas are used as auxiliary decision areas, Area_w is used as the main decision area, and the line segment L be The decision lines are t1 (o1-o2-ba) and t2 (fe-o5-o6) belong to Area_left and Area_right, and are only used as auxiliary stop judgment areas.

[0061] In a specific application embodiment, in order to calibrate the decision area, after installing the visual sensor on the controlled intelligent robot, two parallel lines can be drawn along the left and right sides of the robot. The area between the two lines is actually the road area where the vehicle moves straight ahead. bc and de are the points formed by the two horizontal lines marked at different distances in front of the vehicle according to the actual situation. The effect obtained by collecting a frame of image for each line is as follows: Figure 2 As shown, after obtaining the position information of each calibrated point in the image, the distance of the vehicle relative to the visual sensor and the robot in a stationary or moving state can be obtained based on each point.

[0062] Preferably, the stop decision is configured as the highest priority instruction, that is, in the subsequent motion control process, when the decision instruction is to stop, the other four decision results will be invalid.

[0063] Step S04: Determine the positional relationship between the obstacle area and the decision area, and determine the current action decision based on the judgment result. The action decision includes going straight, turning around, avoiding obstacles, and stopping.

[0064] In this embodiment, the action decisions include five types: forward, stop, U-turn, left obstacle avoidance, and right obstacle avoidance. The action decision to be executed is determined in real time based on the positional relationship between the obstacle area and the decision area. In this embodiment, the current action decision is determined based on the positional relationship between the obstacle area and the decision area, specifically including:

[0065] Determine whether there is an obstacle in the main decision area Area_w. If yes, proceed to determine whether there is an obstacle in the left area Area_left and the right area Area_right. If not, perform a straight-ahead action.

[0066] Determine whether there are obstacles in the left area Area_left and the right area Area_right. If it is determined that there are no obstacles or there is an obstacle in the right area Area_left, perform the left obstacle avoidance action. If it is determined that there is an obstacle in the left area Area_right, perform the right obstacle avoidance action. If it is determined that there are obstacles in both the left area Area_left and the right area Area_right, perform a U-turn.

[0067] In this embodiment, the t1 area and the t2 area located below the decision line in the left area Area_left and the right area Area_right are respectively used as auxiliary stop judgment areas to assist in determining whether to perform a stop action. When it is determined that the area of ​​the unknown area in the main decision area Area_w exceeds the preset threshold, the area of ​​the unknown area in the t1 area and the t2 area is determined. If the area of ​​the unknown area in the t1 area or the t2 area exceeds the preset threshold, the stop action is performed.

[0068] In a specific application embodiment, Figure 4As shown in the figure, after the RGB image is collected by the visual sensor and the RGB image and depth image in front of the controlled intelligent robot are collected by the depth estimation function, the RGB image is segmented and the depth image is clustered to segment the grass area, non-grass area and obstacle area, where the obstacle area is marked in red. First, it is determined whether there is a red pixel in the W area (main decision area Area_w). If not, it is determined to execute a straight action. If so, it is further determined whether there is a red pixel in the L area (left area Area_left) or the R area (right area Area_right). If not, the left obstacle avoidance action is controlled to be executed. If yes, it is further determined whether there are red pixels in both the L area (left area Area_left) and the R area (right area Area_right). If so, a U-turn action is executed. Otherwise, it is determined whether there are red pixels in the L area (left area Area_left). If so, a right obstacle avoidance action is executed, otherwise a left obstacle avoidance action is executed.

[0069] Step S05. Obtain the yaw angle output by the fusion of vision and IMU, calculate the compensation parameters based on the current action decision and the current yaw angle, and send them to the control end of the controlled intelligent mowing robot to control the controlled intelligent mowing robot to execute the current action decision until the full coverage mowing task is completed.

[0070] After determining the action decision required by the controlled intelligent robot in real time, this embodiment calculates compensation parameters based on the yaw angle. This yaw angle is obtained by fusion of vision and IMU technology to improve the robustness of yaw angle acquisition. For example, the yaw angle can be calculated based on the visual image and obtained from the IMU device separately. The yaw angle calculated based on the visual image and the yaw angle obtained by the IMU are then fused to obtain the final yaw angle. Of course, other fusion methods, such as loose coupling or tight coupling, can also be used depending on actual needs. The compensation parameters are then sent to the control terminal of the controlled intelligent mowing robot, forcing it to execute the current action decision. The above steps are repeated until the full coverage mowing task is completed. For example, if the current action decision is to go straight, the compensation parameters for maintaining the straight-line movement are calculated based on the current real-time yaw angle and sent to the control terminal of the controlled intelligent mowing robot, thereby controlling the controlled intelligent mowing robot to maintain the straight-line movement. If the action decision changes, the compensation parameters are recalculated based on the desired action decision and the real-time yaw angle. By combining visual sensors, IMU sensors, and image processing, full coverage control can be achieved without the need for a priori boundary maps and global positioning information. On the one hand, this can solve the cost issue caused by sensors, and on the other hand, it can simulate human behavior to solve the cumbersome problem of constructing a priori boundary maps due to large mowing areas.

[0071] Preferably, a VINS-Mono module can be further used to provide the required yaw information and local short-term two-dimensional positioning information (current x ,current y ), when faced with the loss of visual texture in a large scene, the IMU can be relied upon to provide short-term reliable positioning, that is, when vision is lost, the IO mode is quickly entered to provide pure IMU positioning, avoiding the positioning failure problem caused by pure visual texture loss. The speed drift of SLAM under long-term working conditions will cause the divergence of position and posture. Further, by introducing zero velocity update (ZUPT), the zero velocity correction can be performed immediately after the mowing robot enters a short stop. In this embodiment, the mowing robot is controlled by the control module to complete the full coverage mowing task. The control module receives the decision instructions from the decision module and issues motion commands to the mowing robot according to the corresponding decision instructions.

[0072] In this embodiment, when performing an obstacle avoidance action, the area occupied by the obstacle pixels is obtained, and the steering angle corresponding to the current area occupied by the obstacle pixels is queried from the steering angle database. The controlled intelligent lawn mowing robot is controlled to turn according to the steering angle obtained to complete the obstacle avoidance. The steering angle database stores the required steering angles corresponding to different obstacle pixel occupied areas.

[0073] Traditional obstacle avoidance methods (such as the A* algorithm) require global positioning information and the location information of obstacles in the map to achieve obstacle avoidance. For example, when encountering an obstacle, the A* algorithm needs to provide the current position in the map, the size of the obstacle, and the location information of the next point. This embodiment does not require obtaining a priori maps and positioning information. When encountering an obstacle, there is no need to determine the end position that can bypass the obstacle. Instead, it uses the image in front of the robot captured by the visual sensor to perform regional segmentation, calibrate the robot's traversable area (grass area), and simultaneously calibrate the pixel ratio of various obstacles (non-grass) objects in the image. The pixel ratio is matched one-to-one with the actual control amount that can avoid the obstacle to form a control database with different obstacle pixel occupancy sizes. During the real-time control process, when a straight-line obstacle avoidance action is required, the detected obstacle pixel occupancy size is used to obtain the corresponding control amount in the control database to complete the obstacle avoidance task.

[0074] For example, in the decision area Area_w, there is an obstacle area occupying pixel Pixel1. Through actual testing, it is found that when the steering angle is θ1, it can well bypass the obstacle of this size. When the obstacle occupies pixel Pixel2 and is larger than Pixel1, the steering angle is θ2, which can well bypass this type of obstacle. Similarly, according to the obstacles of different sizes, a steering angle θ required for obstacles of different sizes is constructed. iIf there is a database, in the actual obstacle avoidance process, when the pixel occupancy of an obstacle is greater than Pixel1 but smaller than Pixel2, the obstacle can be avoided by directly using the steering angle θ2 by querying the database.

[0075] In this embodiment, when executing a U-turn, the starting position information of the controlled intelligent mowing robot before the U-turn is recorded, and the robot turns at a specified turning angle from the current position. If the real-time position information of the controlled intelligent mowing robot meets the starting position information, d =|D-(x c -x a )|≤T d , then stop the movement, and then adjust the robot movement with the maximum steering mode and yaw reference benchmark. If the yaw angle e<T e When the conditions are met, the movement stops and the U-turn task is completed, where D represents the full coverage row spacing, x c Indicates the horizontal offset of the position point after the real-time U-turn is completed, x a Indicates the lateral offset of the position point when starting the U-turn, T d represents the deviation threshold, T e represents the yaw deviation threshold. (Xc-Xa) is the lateral distance the robot moves during a U-turn. If (Xc-Xa) approaches the full coverage row spacing, and the difference between the two is less than the set threshold, the robot stops. If it does not stop, it continues straight until the next decision is made. By controlling the robot's motion in this manner, it is possible to achieve full "bow" coverage.

[0076] This embodiment specifically sends control messages to the lower layer in the form of ROS topics. The content of the messages sent mainly includes the robot's movement speed, steering angle, and movement status (forward, backward, stop), etc. After receiving the message, the lower-layer module parses the robot's movement speed, movement status, and steering angle in the message and executes it.

[0077] In this embodiment, a visual sensor is installed in front and rear of the controlled intelligent lawn mower robot, which is suitable for lawn mower robots that cannot turn on the spot. It is understandable that only one visual sensor can be installed in front of the controlled intelligent lawn mower robot, which is suitable for lawn mower robots that can turn on the spot. By combining a single visual sensor and IMU, the "ox plowing method" can be directly adopted to achieve full coverage mowing operation control of the lawn mower robot, which can further reduce hardware costs.

[0078] Example 2:

[0079] This embodiment adopts the control method in embodiment 1 to achieve the following Figure 5The complete process of the full coverage path shown in the figure, where after moving from the starting point S to point a, a U-turn is required to complete the mowing task from point c to point d. The detailed steps are as follows:

[0080] Step 1: The system starts, and there is no obstacle ahead. Receive the forward decision instruction from the decision module. Before starting the movement, record the obtained yaw angle as the reference base angle base_yaw. The yaw angle can be normalized to [-π,π]. Let e ​​= |base_yaw-current_yaw|, if e ≥ T e Then the yaw deviation value is calculated to obtain the compensation parameter, which is sent to the bottom layer of the mowing robot. Through real-time dynamic adjustment, it is ensured that the mowing robot can move in a straight line as much as possible.

[0081] Step 2: When the mowing robot reaches point a and reaches the edge of the lawn, the control module will receive a U-turn command from the decision module. First, record the position information of point a as P a (x a ,y a ), the mowing robot moves slowly at a fixed steering angle and a fixed speed. If the mowing robot is a type that cannot make a U-turn in place, the fixed steering angle must be less than the maximum steering angle of the mowing robot. Determine whether the real-time position information of the mowing robot satisfies the following conditions with point a: d =|D-(x c -x a )|≤T d , where D represents the line spacing between line segments Sa and cd, i.e., the distance that can ensure the complete coverage of the robot (full coverage line spacing), Xc and Xa represent the lateral coordinates of point c and point a respectively. The position information of these two points can be obtained by wheel odometer or visual odometer, etc. When the mowing robot meets the above conditions and reaches point b, it stops moving, and then quickly adjusts the robot movement with the maximum steering mode and yaw reference base_yaw. At this time, the yaw angle e <T e When the conditions are met, the motion stops and the mowing robot reaches point C. The entire mowing robot completes the "U-turn" task. At this time, the direction of the mowing robot actually does not change.

[0082] Step 3: After completing the U-turn task, the robot begins to reverse and mow the lawn. Assuming that the front vision sensor is CAM0 and the rear vision sensor is CAM1, the decision image data for the reverse straight mowing task in the cd segment comes from CAM1, and the decision image data for the Sa segment comes from CAM0. The same process as step 2 is followed to reverse and mow the lawn in a straight line.

[0083] Step 4: When encountering an obstacle, query the steering angle database according to the pixel size of the obstacle to query the required steering angle to set the obstacle avoidance motion angle until the obstacle avoidance is completed.

[0084] Step 5: Repeat steps 1 to 4 above. The robot can Figure 4 The fully covered path shown completes the mowing task, and a parking action is performed after the mowing task is completed.

[0085] Traditional Ackerman-type robotic lawn mowers require a three-stage reverse curve to achieve a U-turn, which can affect the robot's efficiency. This embodiment, by controlling the robot to mow in reverse, eliminates the need for a U-turn, effectively improving its efficiency to a certain extent.

[0086] The present invention uses only visual sensors combined with IMU to achieve full-coverage operation control of the lawn mowing robot without relying on any global positioning system and without building any prior map. It does not require a large amount of action training and reasoning testing, which can greatly reduce the implementation cost and control complexity while ensuring control accuracy and reliability.

[0087] This embodiment further provides an intelligent lawn mowing robot device, comprising a processor and a memory, wherein the memory is used to store a computer program, and the processor is used to execute the computer program to perform the above method.

[0088] It is understandable that the above method of this embodiment can be executed by a single device, such as a computer or server, etc., and can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In the case of a distributed scenario, one of the multiple devices can only execute one or more steps in the above method of this embodiment, and multiple devices interact to complete the above method. The processor can be implemented in the form of a general-purpose CPU, a microprocessor, an application-specific integrated circuit, or one or more integrated circuits, etc., for executing relevant programs to implement the above method of this embodiment. The memory can be implemented in the form of a read-only memory ROM, a random access memory RAM, a static storage device, and a dynamic storage device. The memory can store an operating system and other application programs. When the above method of this embodiment is implemented by software or firmware, the relevant program code is stored in the memory and called and executed by the processor.

[0089] This embodiment further provides a computer-readable storage medium storing a computer program, which implements the above method when executed by a processor.

[0090] Those skilled in the art will appreciate that the above-mentioned embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code. The present invention is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, may be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the functions described in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing device to work in a specific way, so that the instructions stored in the computer-readable memory produce a product including the instruction device, which implements the function specified in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 These computer program instructions can also be loaded onto a computer or other programmable data processing device, so that a series of operation steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0091] The above description is merely a preferred embodiment of the present invention and does not constitute any form of limitation to the present invention. Although the present invention has been disclosed above with reference to the preferred embodiment, it is not intended to limit the present invention. Therefore, any simple modifications, equivalent variations, and modifications to the above embodiment that do not depart from the technical solution of the present invention and are based on the technical essence of the present invention shall fall within the scope of protection of the technical solution of the present invention.

Claims

1. A method for controlling a non-mapped full-coverage operation of an intelligent lawn mowing robot, applied to an intelligent lawn mowing robot equipped with at least one visual sensor and at least one IMU, wherein the visual sensor is arranged at the forward direction end of the intelligent lawn mowing robot, characterized in that: The method steps include: Controlling the controlled intelligent lawn mower robot to start moving from a starting position, and obtaining an image of the controlled intelligent lawn mower robot's moving direction collected by a visual sensor during the moving process of the controlled intelligent lawn mower robot; The currently acquired image is segmented into grass and non-grass areas using a segmentation model, and obstacles are identified, grass and non-grass areas are segmented, and obstacle areas are marked; Determining a decision area in the currently acquired image according to the width of the controlled intelligent lawn mowing robot and the segmented grass area, wherein the decision area corresponds to the forward movement area of ​​the controlled intelligent lawn mowing robot in the image; Determine the positional relationship between the obstacle area and the decision area, and determine a current action decision based on the determination result, wherein the action decision includes going straight, turning around, avoiding obstacles, and stopping; Obtain the yaw angle output by the fusion of vision and IMU, calculate the compensation parameter based on the currently obtained action decision and the currently obtained yaw angle, and send it to the control end of the controlled intelligent mowing robot to control the controlled intelligent mowing robot to execute the current action decision until the full coverage mowing task is performed.

2. The method for controlling the full coverage operation of an intelligent lawn mowing robot without a map according to claim 1, characterized in that: The left and right boundary lines of the decision area correspond to the left and right reference trajectory lines of the controlled intelligent lawn mowing robot in the forward direction, respectively. The reference trajectory lines extend from the end close to the controlled intelligent lawn mowing robot in the image as the starting point toward the forward direction of the controlled intelligent lawn mowing robot for a specified length, and the distance between the two reference trajectory lines is greater than the width of the controlled intelligent lawn mowing robot.

3. The method for controlling the non-mapped full coverage operation of the intelligent lawn mowing robot according to claim 2, characterized in that: It also includes setting a decision line parallel to the bottom of the image according to the maximum steering angle of the controlled intelligent lawn mowing robot, dividing the area above the decision line in the decision area into a main decision area Area_w, and the height of the decision line from the bottom of the image is configured so that the controlled intelligent lawn mowing robot does not touch the obstacle in front when moving at the maximum steering angle. Whether to perform a straight-line action is determined based on whether there is an obstacle in the main decision area Area_w, and the left area Area_left and the right area Area_right are used as stop auxiliary judgment areas to assist in determining whether to perform a U-turn or obstacle avoidance action. The left area Area_left is the area between the left reference trajectory line and the left boundary of the image, and the right area Area_right is the area between the right reference trajectory line and the right boundary of the image.

4. The method for controlling the non-mapped full coverage operation of the intelligent lawn mowing robot according to claim 3, characterized in that: The determining of the positional relationship between the obstacle area and the decision area, and determining the current action decision according to the determination result includes: Determine whether there is an obstacle in the main decision area Area_w. If yes, proceed to determine whether there is an obstacle in the left area Area_left and the right area Area_right. If not, perform a straight-ahead action. Determine whether there are obstacles in the left area Area_left and the right area Area_right. If it is determined that there are no obstacles or there is an obstacle in the right area Area_left, perform the left obstacle avoidance action. If it is determined that there is an obstacle in the left area Area_right, perform the right obstacle avoidance action. If it is determined that there are obstacles in both the left area Area_left and the right area Area_right, perform a U-turn.

5. The method for controlling the non-mapped full coverage operation of the intelligent lawn mowing robot according to claim 4, characterized in that: It also includes using the t1 area and t2 area located below the decision line in the left area Area_left and the right area Area_right as auxiliary stop judgment areas to assist in determining whether to execute the stop action. When it is determined that the area of ​​the unknown area in the main decision area Area_w exceeds the preset threshold, the area of ​​the unknown area in the t1 area and the t2 area is determined. If the area of ​​the unknown area in the t1 area or the t2 area exceeds the preset threshold, the stop action is executed.

6. The method for controlling the non-mapped full coverage operation of an intelligent lawn mowing robot according to any one of claims 1 to 5, characterized in that: When performing an obstacle avoidance action, the area occupied by the obstacle pixels is obtained, and the steering angle corresponding to the current area occupied by the obstacle pixels is queried from the steering angle database. The controlled intelligent lawn mower robot is controlled to turn according to the steering angle obtained to complete the obstacle avoidance. The steering angle database stores the required steering angles corresponding to different obstacle pixel occupied areas.

7. The method for controlling the non-mapped full coverage operation of an intelligent lawn mowing robot according to any one of claims 1 to 5, characterized in that: When executing the U-turn action, the starting position information of the controlled intelligent mowing robot before the U-turn is recorded, and the robot turns at the specified turning angle from the current position. If the real-time position information of the controlled intelligent mowing robot meets the starting position information, d =|D-(x c -x a |≤T d , then stop the movement, and then adjust the robot movement with the maximum steering mode and yaw reference benchmark. If the yaw angle e<T e When the conditions are met, the movement stops and the U-turn task is completed, where D represents the full coverage row spacing, x c Indicates the horizontal offset of the position point after the real-time U-turn is completed, x a Indicates the lateral offset of the position point when starting the U-turn, T d represents the deviation threshold, T e Indicates the yaw deviation threshold.

8. The method for controlling the non-mapped full coverage operation of an intelligent lawn mowing robot according to any one of claims 1 to 5, characterized in that: One or more visual sensors are installed in front and rear of the controlled intelligent lawn mower robot. When the controlled intelligent lawn mower robot is traveling along the forward path, the image captured by the front visual sensor is obtained to determine the current action decision. After completing the mowing task of the forward path, when it needs to return to perform the mowing task of the backward path, the controlled intelligent lawn mower robot is controlled to travel in a reverse motion mode, and when the controlled intelligent lawn mower robot is traveling backward, the image captured by the rear visual sensor is obtained to determine the current action decision.

9. An intelligent lawn mowing robot device, comprising a processor and a memory, wherein the memory is used to store a computer program, characterized in that: The processor is configured to execute the computer program to perform the method according to any one of claims 1 to 8.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images