Camera-based obstacle, boundary detection method and automatic walking device

Abstract

The application provides a camera-based obstacle and boundary detection method and an automatic walking device. The image acquisition unit on the automatic walking device is used to collect the environmental image in front of the running direction in real time, and then the environmental image is rasterized according to different coordinate intervals, and the non-grassland information in each grid area is detected, so that the automatic walking device adjusts the running direction according to the non-grassland information and the distance characteristics of the grid area to which the non-grassland information belongs. The application uses the local environmental image information of the automatic walking device, and can identify the grassland boundary through simple processing, which can effectively avoid the misoperation of the airborne monocular camera caused by the lack of depth data after boundary detection, and can effectively avoid the substantial increase of algorithm cost or computing platform cost, can reduce the transmission amount of the boundary detection result and the corresponding depth data, and facilitate the machine chassis control system to control the machine running track or safety action.

Description

Technical Field

[0001] This application relates to the field of garden tools, and more specifically to a camera-based obstacle and boundary detection method and an automated walking device. Background Technology

[0002] Currently, automatic lawnmowers are divided into two types: those with boundaries and those without. The key difference lies in whether or not manual installation of lawn boundary signal lines is required. Manually laying electromagnetic coils to create lawn boundary signal lines typically incurs high labor and time costs, and has therefore been gradually replaced by borderless mowing technology. Borderless mowing technology usually utilizes outdoor positioning technology to define a virtual boundary for the lawn, thereby driving the automatic walking device to operate only within the area enclosed by this virtual boundary based on its real-time coordinate position.

[0003] Both of the above methods are susceptible to failure in boundary detection or outdoor positioning systems, which could directly cause the machine to run off the grass boundary. Therefore, regardless of the type of automatic lawnmower, a technical solution is needed that can detect the working area boundary based on local device data. Summary of the Invention

[0004] This application addresses the shortcomings of existing technologies by providing a camera-based obstacle and boundary detection method and an automated walking device. This application pre-calculates the distance features corresponding to different coordinate intervals in the environmental image of the automated walking device, thereby directly responding to non-grassland information within different coordinate intervals locally collected by the automated walking device. This enables obstacle avoidance or allows the device to turn around at grass boundaries and continue traversing the work area. The specific technical solution adopted in this application is as follows.

[0005] First, to achieve the above objectives, a camera-based obstacle and boundary detection method is proposed. The steps include: acquiring real-time environmental images in front of the autonomous walking device's running direction; rasterizing the environmental images according to different coordinate intervals; detecting non-grass information in the environmental images within each rasterized grid area; and triggering the autonomous walking device to adjust its running direction based on the distance characteristics of the non-grass information and its corresponding grid area. The distance characteristics corresponding to each grid area are pre-calculated based on the structure of the autonomous walking device.

[0006] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, the step of rasterizing the environmental image includes: dividing the lower part of the environmental image into several raster regions according to preset horizontal and vertical coordinate intervals, respectively, along directions parallel to the horizontal and vertical edges of the environmental image.

[0007] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, the step of detecting non-grass information in the environmental image within each grid area includes: performing mean-shift filtering on the image; identifying non-preset color ranges based on color information in the filtered image; and using a continuous segmentation method to extract the boundary lines of the grass corresponding to the non-preset color ranges and mark the non-grass information.

[0008] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, the step of triggering the automatic walking device to adjust its running direction based on the distance characteristics of the non-grass information and its corresponding grid area includes: querying the grid area surrounding the non-grass information and the non-grass information contained in several frames of images before and after; and driving the automatic walking device to adjust its running direction to avoid the running direction corresponding to the grid area containing the non-grass information.

[0009] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, during the process of triggering the automatic walking device to adjust its running direction based on the non-grass information and the distance characteristics of the grid area to which it belongs, the non-grass information contained in the grid area located at the bottom edge of the environmental image is responded to first, and the non-grass information contained in the grid area located at the top of the environmental image is responded to last.

[0010] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, when avoiding the running direction corresponding to the grid area containing non-grass information, the following steps are specifically performed: driving the automatic walking device to deflect towards other grid areas that do not contain non-grass information; or driving the automatic walking device to turn around when the grid area containing non-grass information laterally crosses the entire environmental image.

[0011] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, when avoiding the running direction corresponding to the grid area containing non-grass information, the following steps are specifically performed: when the distance feature corresponding to the grid area does not reach the preset standard, continue to drive the automatic walking device along the original running direction; when the distance feature corresponding to the grid area reaches the preset standard, drive the automatic walking device to deflect or turn around to other grid areas that do not contain non-grass information.

[0012] Optionally, in any of the above-described camera-based obstacle and boundary detection methods, the distance features corresponding to each grid region are pre-calculated as follows: based on the shooting height h of the image acquisition unit on the automated walking device, the vertical field of view β of the image acquisition unit, and the field of view α corresponding to the safe distance L2 in front of the automated walking device, the environmental image is divided into an upper part outside the safe distance L2 in front of the device and a lower part within the safe distance L2 in front of the device; the average distance of each pixel position in the grid region from the automated walking device is calculated according to the field of view α, or the distance of any pixel position in each grid region from the automated walking device is used as the distance feature corresponding to that grid region.

[0013] Meanwhile, to achieve the above objectives, this application also provides an automatic walking device, which includes: a self-propelled drive system for driving the automatic walking device to run and adjusting its running direction; an image acquisition unit disposed on the casing of the automatic walking device for acquiring environmental images in front of the automatic walking device in its running direction; and a control unit connected to the self-propelled drive system and the image acquisition unit for executing any of the above methods based on the environmental images in front of the automatic walking device in its running direction acquired in real time by the image acquisition unit, thereby triggering the self-propelled drive system to adjust the running direction of the automatic walking device.

[0014] Optionally, in any of the above-described automatic walking devices, the image acquisition unit employs a depth camera, and the distance features corresponding to each grid area are pre-calculated as follows: the average distance between each depth detection point in the grid area and the depth camera's shooting point is calculated, or the distance between any pixel position in each grid area and the automatic walking device is used as the distance feature corresponding to that grid area. Beneficial effects

[0015] This application provides a camera-based obstacle and boundary detection method and an automated walking device. It utilizes the image acquisition unit on the automated walking device to acquire real-time environmental images in the direction of travel. Then, it rasterizes the environmental images according to different coordinate intervals, detecting non-grass information within each raster region. Based on the non-grass information and the distance characteristics of its respective raster region, the automated walking device is triggered to adjust its direction of travel. This application utilizes the local environmental image information of the automated walking device, and through simple processing, it can identify grass boundaries. This effectively avoids the malfunctions caused by the lack of depth data after boundary detection by airborne monocular cameras, and also effectively avoids the significant increase in algorithm or computing platform costs caused by depth estimation by monocular cameras. This application directly estimates the boundary detection results and corresponding estimated depth in the current image frame through raster regional encoding, which reduces the amount of data transmitted for boundary detection results and corresponding depth data, facilitating the machine chassis control system's control of machine trajectory or safety procedures.

[0016] Other features and advantages of this application will be set forth in the following description, and will be apparent in part from the description, or may be learned by practicing this application. Attached Figure Description

[0017] The accompanying drawings are provided to further illustrate the present application and form part of the specification. Together with the embodiments of the present application, they serve to explain the present application but do not constitute a limitation thereof. In the drawings:

[0018] Figure 1 This is a schematic diagram showing the installation location and field of view distribution of the image acquisition unit on the automated walking device of this application;

[0019] Figure 2 yes Figure 1 A schematic diagram showing the distribution of the safe distance range in front of the equipment in the environmental images collected in the image;

[0020] Figure 3 This is a schematic diagram of the environmental image rasterization process in this application;

[0021] Figure 4 A schematic diagram showing the running direction corresponding to each grid area from the perspective of an automated walking device;

[0022] Figure 5 This is a schematic diagram of the boundary line detection output.

[0023] In the figure, 101 represents the image acquisition unit; 102 represents the automatic walking device; 103 represents the distance output position point after the image result is rasterized; 104 represents the center point of the raster; and 105 represents the installation position of the image acquisition unit on the automatic walking device. Implementation

[0024] To make the objectives and technical solutions of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this application. All other embodiments obtained by those skilled in the art based on the described embodiments of this application without creative effort are within the scope of protection of this application.

[0025] Those skilled in the art will understand that, unless otherwise defined, all terms used herein (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains. It should also be understood that terms such as those defined in general dictionaries should be understood to have the same meaning as in the context of the prior art, and should not be interpreted in an idealized or overly formal sense unless defined as herein.

[0026] The terms "left" and "right" as used in this application refer to the direction of movement of the automatic walking device, with the left side of the automatic walking device being left and the right side being right, rather than a specific limitation on the device mechanism of this application.

[0027] The term "connection" as used in this application can mean a direct connection between components or an indirect connection between components through other components.

[0028] The terms "up" and "down" as used in this application refer to the direction from the ground to the image acquisition unit when the user is facing the direction of the automatic walking device's movement, which is up, and vice versa, and are not a specific limitation on the device mechanism of this application.

[0029] With the maturity of object detection technology and the decrease in computing costs, this application introduces a camera-based grass boundary detection method into the intelligent lawnmower control system. This method can identify grass boundaries as stripes or blocks in a single image, thereby identifying the location of the boundary and its corresponding distance in the current image, and adjusting the operation of the automated walking device.

[0030] Figure 1 An automatic walking device according to this application is provided with:

[0031] A self-propelled drive system, including walking wheels, corresponding walking motors, and steering mechanisms, is used to drive the automatic walking equipment and adjust its direction of travel.

[0032] The image acquisition unit is installed on the casing of the automated walking equipment, usually at the top of the casing towards the front of the direction of travel, and is used to acquire environmental images in front of the automated walking equipment in the direction of travel.

[0033] The control unit, which is connected to the self-propelled drive system and the image acquisition unit, is used to identify non-grass information in the environmental image in front of the self-propelled device's running direction in real time based on the environmental image in front of the running direction of the self-propelled device acquired by the image acquisition unit in real time, and to trigger the self-propelled drive system to adjust the running direction of the self-propelled device in accordance with the following method steps.

[0034] The image acquisition unit acquires real-time images of the environment in front of the automatically moving equipment in its direction of travel.

[0035] The environmental image is rasterized according to different coordinate intervals. During the processing, the lower part of the environmental image can generally be divided into several raster regions according to the preset horizontal and vertical coordinate intervals, respectively, along the directions parallel to the horizontal and vertical edges of the environmental image.

[0036] Non-grassland information in the environmental image within each raster region obtained after rasterization is detected separately;

[0037] The automatic walking device is triggered to adjust its running direction based on the distance characteristics of non-grass information and its corresponding grid area. For example, the grid area around the non-grass information and the non-grass information contained in several frames of images before and after are queried, and the automatic walking device is driven to adjust its running direction to avoid the running direction corresponding to the grid area containing non-grass information. For example, the automatic walking device is driven to deflect to other grid areas that do not contain non-grass information, or the automatic walking device is driven to turn around when the grid area containing non-grass information crosses the entire environmental image laterally.

[0038] In the above steps, the distance characteristics corresponding to each grid area can be pre-calculated based on the structure of the automated walking device as follows:

[0039] Reference first Figure 1 Based on the shooting height h of the image acquisition unit on the automated walking device, the vertical field of view β of the image acquisition unit, and the field of view α corresponding to the safe distance L2 in front of the automated walking device, the environmental image is segmented into... Figure 2 The upper part of the equipment beyond the safety distance L2 in front of the equipment and the lower part of the equipment within the safety distance L2 in front of the equipment;

[0040] Then calculate according to the field of view angle α. Figure 3 The average distance of each pixel position within each grid area formed by segmentation from the automatic walking device, or the distance of any pixel position within each grid area from the automatic walking device, is used as the distance feature corresponding to that grid area.

[0041] Therefore, this application can, based on a monocular RGB camera or a depth camera, bind the ground safety distance corresponding to different grid regions in the environmental image using information such as the camera's installation angle and field of view, and initialize the parameters of the camera's field of view and the estimated distance of the involved boundaries with the Eulerian distance in the current image frame. Then, the boundary information detected in the real-time acquired current video frame is rasterized and matched, the presence or absence of a boundary in each grid is binarized, and the estimated depth is replaced with the distance from the center point of the rectangle where the grid is located, simplifying the output results.

[0042] In specific applications, this application can use an RGB camera or a combination sensor of RGB + depth camera as the image acquisition unit. The image acquisition unit can be... Figure 4 The method is set on automated walking equipment such as lawnmowers, thereby realizing grassland boundary detection and outputting detection results based on the local camera of the automated walking equipment.

[0043] When specifically performing grassland boundary detection, the camera's vertical field of view β angle and the camera's installation height can be used to pre-calculate the camera's viewing angle α angle corresponding to the required alarm safety distance range L2. The pixel area corresponding to the width of the grassland in front of the lawnmower in the environmental image can then be marked as... Figure 1 A and Figure 2 In the diagram, A represents the safe distance range L2 within the current camera field of view, and the response is calculated focusing on the image region within the L2 range of the captured grassland:

[0044] During the real-time operation of the lawnmower, the field of view of the camera mounted on the lawnmower is rasterized using pixel coordinates to segment the aforementioned region A. Figure 4 The diagram shows several grid regions. Therefore, a boundary detection algorithm can be used to determine the pixel coordinate distribution within each grid cell to identify whether there is a grass boundary.

[0045] If a boundary is detected, the grid is considered to be in a boundary state (assumed to be 1); if no boundary is detected, the grid state is set to 0.

[0046] Then, based on the hardware conditions of the image acquisition unit in the lawnmower, when using a monocular camera, the distance mapping relationship between the calibrated camera pixel area and the front end of the device is determined according to the pre-defined angle and distance information such as the camera height, camera pitch angle, and angle with the horizontal field of view axis, thus clarifying the grass boundary depth. When using an RGB-D camera, an average depth can be output as the distance between the grass boundary and the front end of the device by calculating the average depth of the depth map corresponding to the strip or block images formed by the boundary.

[0047] At this point, each grid can output two types of data: whether a boundary has been detected and the distance from the boundary to the camera. Therefore, based on the detection status of the grass boundary line (i.e., non-grass information) within each grid and the distance characteristics corresponding to that grid, the automatic walking device can be triggered to continue traveling in its original direction if the distance characteristics of the corresponding grid area do not meet a preset standard; otherwise, it will be driven to detour or turn around to other grid areas that do not contain non-grass information when the distance characteristics of the corresponding grid area meet the preset standard. To improve the accuracy of the response to non-grass information such as grass boundary lines, this application, when determining whether obstacle avoidance is necessary, preferably judges whether non-grass information such as grass boundary lines appears continuously based on environmental images within several consecutive frames or a time period. If they appear continuously, the device will avoid the direction of travel corresponding to the grid area containing non-grass information based on the spatial position of the specific grid it corresponds to.

[0048] During the above detection process, when detecting grassland boundaries, non-grassland information in the environmental image within each grid area can be obtained through the following steps:

[0049] Apply mean-shift filtering to the image;

[0050] Based on the color information in the filtered image, non-preset color ranges are identified, and green grass is divided into grass and non-grass regions. The boundary between grass and non-grass is the boundary to be detected.

[0051] Using a continuous segmentation method, the boundary lines of the grassland corresponding to the non-preset color range are extracted, and non-grass information is marked.

[0052] In the above steps, mean-shift filtering can be implemented using a function similar to the pyrMeanShiftFiltering function in OpenCV. The main purpose of using mean-shift filtering is that simple color-based segmentation can be affected by small non-green objects on the grass, such as twigs, leaves, and small patches of withered grass, leading to false detections. Mean-shift filtering neutralizes colors with similar color distributions, smooths color details, erodes smaller color areas, and avoids interference from small color patches. Morphological algorithms can also be used in subsequent image processing to avoid interference from these small color patches.

[0053] After detecting non-grassland boundaries, the automatic walking device can be further triggered to adjust its running direction based on the non-grassland information and the distance characteristics of its corresponding grid region. During this process, since the entity positions corresponding to pixel regions near the bottom edge of the environmental image are closer to the automatic walking device's body, the grid state encoding output can preferably be set according to a principle of from near to far and from left to right. This allows the automatic walking device to prioritize responding to non-grassland information contained in grid regions located at the bottom edge of the environmental image, or to prioritize using the 20% of data closest to the device, and finally responding to non-grassland information contained in grid regions at the top of the environmental image. For example, according to... Figure 5 The output grid coordinates corresponding to the diagonal boundaries of obstacles, shown in the sequence of b1, b2, c2, c3, and d3, ensure that the machine responds first to the obstacle boundary information that is closest to it, determines the actual position of the obstacle, and promptly performs avoidance and response to nearby obstacles.

[0054] In this step, the size of each grid can be flexibly configured according to the detection accuracy or the distance between the obstacle avoidance path and the obstacle. When it is necessary to get closer to the obstacle for obstacle avoidance and avoid making it impossible to work on the grass near the obstacle, the size of each grid can be set smaller. This allows the spatial position of the obstacle to be mapped using the coordinate position of the grid in the image, and the division area formed by the grid can be used to drive the automatic walking device to avoid the obstacle according to the grid area that is closer to the actual position of the obstacle. Using the above principle to output the grass boundary data ensures that the machine first detects and responds to the obstacle closest to the machine, and determines the next action, such as retreating, turning left, or turning right, based on the position of the detected obstacle, thereby avoiding the obstacle.

[0055] Considering the usage scenarios, the grass in environments such as courtyards, gardens, and golf courses is generally flat and gently sloping. Therefore, when pre-binding the mapping relationship between different pixel coordinate regions in the environmental image and the ground in front of the automated walking device in this application, the influence of ground undulations on the mapping distance relationship does not need to be considered.

[0056] In practice, the distance characteristics corresponding to each grid area can be determined by the average distance between each depth detection point within the grid area and the depth camera's shooting point, or the distance between any pixel position within the grid area and the automated walking device, or by using the distances corresponding to fixed points in each grid obtained through geometric calculations based on the center point of the grid area or any image coordinates, according to the camera's set height and shooting angle. Furthermore, the viewing angle range corresponding to each grid area and its approximate distance from the bottom edge of the automated walking device can be clearly defined by the grid's numbering from bottom to top. Therefore, after determining the grid division size and the pixel positions corresponding to each grid, obstacle avoidance or movement along the boundary line can be quickly triggered based on whether the work area boundary or non-grass boundary is detected within each grid area.

[0057] The above are merely embodiments of this application, and their descriptions are quite specific and detailed, but they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application.

Claims

1. A camera-based obstacle and boundary detection method, characterized in that the steps include... include: Acquire real-time environmental images of the area in front of the autonomous walking device in its direction of travel; Rasterization of environmental images according to different coordinate ranges; Non-grassland information in the environmental image within each raster region obtained after rasterization is detected separately; The automatic walking device is triggered to adjust its running direction based on non-grassland information and the distance characteristics of its corresponding grid area; The distance characteristics corresponding to each grid area are obtained in advance based on the structure of the automated walking equipment. The steps for detecting non-grassland information in the environmental image within each grid region include: First, the image is subjected to mean-shift filtering, and then the non-preset color range is identified based on the color information in the filtered image. Finally, the continuous segmentation method is used to extract the boundary lines of the grassland corresponding to the non-preset color range and mark the non-grass information; The steps for triggering the automatic walking device to adjust its running direction based on non-grassland information and the distance characteristics of its corresponding grid area include: Query the raster region surrounding non-grassland information and the non-grassland information contained in several frames before and after it; The automatic walking device is driven to adjust its running direction to avoid the running direction corresponding to the grid area containing non-grassland information. Specifically, when avoiding the running direction corresponding to the grid area containing non-grassland information, the following steps are performed: If the distance feature corresponding to the grid area does not meet the preset standard, the automatic walking device will continue to travel in the original direction of operation. When the distance characteristics corresponding to the grid area reach the preset standard, the automatic walking device is driven to deflect or turn around to other grid areas that do not contain non-grassland information. In the above steps, the distance features corresponding to each grid region are pre-calculated as follows: Based on the shooting height h of the image acquisition unit on the automatic walking device, the vertical field of view β of the image acquisition unit, and the field of view α corresponding to the safe distance L2 in front of the automatic walking device, the environmental image is divided into an upper part outside the safe distance L2 in front of the device and a lower part inside the safe distance L2 in front of the device. The average distance between each pixel position within the grid area and the automatic walking device is calculated based on the field of view angle α, or the distance between any pixel position within each grid area and the automatic walking device is used as the distance feature corresponding to that grid area.

2. The obstacle and boundary detection method based on a camera as described in claim 1, characterized in that, The steps for rasterizing environmental images include: The lower part of the environment image is divided into several grid areas according to the preset horizontal and vertical coordinate spacing, respectively, along the directions parallel to the horizontal and vertical edges of the environment image.

3. The obstacle and boundary detection method based on a camera as described in claim 2, characterized in that, During the process of triggering the automatic walking device to adjust its running direction based on the distance characteristics of non-grassland information and its corresponding grid area, the device prioritizes responding to non-grassland information contained in the grid area located at the bottom edge of the environmental image, and finally responds to non-grassland information contained in the grid area located at the top of the environmental image.

4. The obstacle and boundary detection method based on a camera as described in claim 3, characterized in that, When avoiding the running direction corresponding to the grid area containing non-grassland information, the following steps are specifically performed: Drive the automated walking device to deflect towards other grid areas that do not contain non-grassland information; Alternatively, the autonomous walking device may be driven to turn around when the grid area containing non-grassland information traverses the entire environmental image laterally.

5. An automatic walking device, characterized in that, The automated walking device is equipped with: Self-propelled drive system, used to drive automatic walking equipment and adjust its direction of travel; An image acquisition unit is installed on the housing of the automated walking device and is used to acquire environmental images in front of the automated walking device in the direction of operation. The control unit, which is connected to the self-propelled drive system and the image acquisition unit, is used to execute the method described in any one of claims 1 to 4 based on the environmental image in front of the self-propelled device that is acquired in real time by the image acquisition unit, thereby triggering the self-propelled drive system to adjust the running direction of the self-propelled device.

6. The automatic walking device as described in claim 5, characterized in that, The image acquisition unit uses a depth camera, and the distance features corresponding to each grid area are pre-calculated as follows: the average distance between each depth detection point in the grid area and the depth camera shooting point, or the distance between any pixel position in each grid area and the automatic walking device, is used as the distance feature corresponding to that grid area.

Images