Multi-sensor obstacle recognition method, system and terminal for lawn mowing robots

Abstract

This invention provides a multi-sensor obstacle recognition method, system, and terminal for lawnmower robots. The lawnmower integrates a visual sensor and a time-of-flight (TOF) sensor. By acquiring images and TOF data, the system processes the images and extracts a mask for the grass region. Then, based on the images and TOF data, point cloud partitioning and grass height estimation are performed. Combined with the grass region mask, accurate obstacle recognition is achieved. This method combines the advantages of visual and TOF sensing technologies to construct a more robust and efficient obstacle recognition system, significantly improving the lawnmower robot's adaptability in complex environments and ensuring its operational safety. This invention effectively reduces the limitations of lawnmower usage, significantly reduces the missed and false detections of small obstacles due to grass obstruction, and avoids misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition.

Description

Technical Field

[0001] This invention relates to the field of lawnmower technology, and in particular to a method, system and terminal for multi-sensor obstacle recognition in lawnmowers. Background Technology

[0002] With the rapid development of smart technologies, automated equipment is being used more and more widely in daily life. Especially in agriculture and horticulture, automated lawnmowers are gradually becoming important tools for improving work efficiency and reducing labor costs. These robots are typically equipped with multiple sensors to achieve autonomous navigation and environmental awareness, enabling them to work safely and efficiently in complex outdoor environments.

[0003] In lawnmowers, visual sensors and time-of-flight (TOF) sensors are two common sensor types. Visual sensors use cameras to capture images of the surrounding environment and employ image processing techniques for obstacle recognition and path planning. The advantage of visual sensors lies in their high resolution and rich color information, enabling them to effectively identify different types of obstacles, such as trees, rocks, and other gardening fixtures. However, the performance of visual sensors can be limited under extreme conditions such as strong light, shadow, and low light, and they may be unable to accurately acquire 3D information about obstacles, leading to decreased recognition accuracy.

[0004] On the other hand, Time-of-Flight (TOF) sensors acquire depth information by emitting laser pulses and measuring their return time, providing high-precision distance measurements. The working principle of TOF sensors allows them to maintain good performance in environments with varying lighting conditions, making them suitable for obstacle identification and localization. This makes TOF sensors play an important role in lawnmower robots, especially in identifying complex terrain and irregular obstacles. However, due to the lack of information such as color and texture, it is often impossible to determine whether the obstacle identified by TOF is accurate. Summary of the Invention

[0005] In view of the shortcomings of the prior art described above, the purpose of this invention is to provide a multi-sensor obstacle recognition method, system and terminal for lawnmower robots, which solves the problems of limited use of lawnmowers when using a single sensor for obstacle recognition, the problem of missed or false detection of small obstacles caused by grass occlusion, and the misidentification of tall grass and weeds in the grass.

[0006] To achieve the above and other related objectives, the present invention provides a multi-sensor obstacle recognition method for a lawnmower robot. The method includes: acquiring images and TOF data collected by a visual sensor and a time-of-flight ranging sensor installed on the lawnmower robot; processing the images and extracting a grass area mask; performing point cloud partitioning and grass height estimation based on the images and TOF data, and recognizing obstacles by combining the extracted grass area mask.

[0007] In one embodiment of the present invention, the step of processing the image data and extracting the grassland region mask includes: performing distortion processing and image region segmentation on the acquired image to obtain a segmented image; performing Gaussian filtering on the segmented image to obtain a filtered image; converting the filtered image from the RGB color gamut to the HSV color gamut to obtain an HSV image; segmenting the HSV image based on the hue threshold obtained from the grassland test to obtain a preliminary grassland region mask; performing erosion and dilation operations on the non-grass regions in the preliminary grassland region mask to obtain a processed mask; extracting the boundaries of the non-grass regions in the processed mask and calculating the region area to extract the final grassland region mask.

[0008] In one embodiment of the present invention, the step of performing distortion processing and image region segmentation on the acquired image to obtain a segmented image includes: performing distortion correction processing on the image based on calibrated image intrinsic parameters; and segmenting the lower half of the distortion-corrected image containing the grassland region as the segmented image.

[0009] In one embodiment of the present invention, the step of extracting the boundaries of non-grassland areas in the processed mask and calculating the area of ​​the areas to extract the final grassland area mask includes: extracting all closed boundaries of non-grassland areas in the processed mask to determine each sub-region, and calculating the minimum bounding rectangle area of ​​each sub-region; based on a set area threshold, filtering out small-area noise in the processed mask according to the minimum bounding rectangle area of ​​each sub-region to obtain the final grassland area mask.

[0010] In one embodiment of the present invention, the step of performing point cloud partitioning and grass height estimation based on the image and TOF data, and performing obstacle recognition in combination with the extracted grass region mask includes: temporally aligning the acquired image with the TOF data; identifying point clouds in the aligned TOF data that can be projected onto the aligned image as common-view point clouds and point clouds that cannot be projected onto the aligned image as non-common-view point clouds; performing voxel filtering downsampling on the common-view point clouds to obtain processed common-view point clouds; calculating the adaptive grass height of the processed common-view point clouds and determining the grass height range of the common-view point clouds and non-common-view point clouds for point cloud filtering; performing density clustering and point cloud cluster obstacle judgment on the filtered common-view point clouds based on the grass region mask to obtain obstacle point clouds in the common-view area; performing density clustering and point cloud cluster obstacle judgment on the filtered non-common-view point clouds to obtain obstacle point clouds in the non-common-view area; and combining the common-view and non-common-view obstacle point clouds to obtain the final obstacle point cloud.

[0011] In one embodiment of the present invention, the step of calculating the adaptive grass height of the processed common-view point cloud and determining the grass height range of the common-view point cloud and the non-common-view point cloud for point cloud filtering includes: statistically analyzing the height morphology distribution of the processed common-view point cloud, calculating the adaptive grass height of the common-view area, and filtering out point clouds in the processed common-view point cloud that are not higher than the adaptive grass height; calculating the target grass height of the non-common-view area based on the adaptive grass height of the common-view area and a preset threshold, and filtering out point clouds in the processed non-common-view point cloud that are not higher than the target grass height.

[0012] In one embodiment of the present invention, the step of performing density clustering and obstacle determination on the filtered common view area point cloud based on the grassland area mask to obtain the obstacle point cloud of the common view area includes: performing density clustering on the filtered common view area point cloud to obtain one or more point cloud clusters of the common view area; and determining the obstacle point cloud of the common view area based on the position of the point cloud cluster projected onto the grassland area mask and the height morphology distribution of the point cloud cluster.

[0013] In one embodiment of the present invention, the step of performing density clustering and obstacle judgment on the filtered non-common view point cloud to obtain obstacle point cloud in the non-common view includes: performing density clustering on the filtered non-common view point cloud to obtain point cloud clusters in the non-common view; and judging obstacle point cloud in the non-common view based on the morphological distribution of the point cloud clusters.

[0014] To achieve the above and other related objectives, the present invention provides a multi-sensor obstacle recognition system for a lawnmower robot. The system includes: a data acquisition module for acquiring images and TOF data collected by a visual sensor and a time-of-flight ranging sensor mounted on the lawnmower robot; a grass region mask extraction module connected to the data acquisition module for processing the images and extracting grass region masks; and an obstacle recognition module connected to the data acquisition module and the grass region mask extraction module for performing point cloud partitioning and grass height estimation based on the images and TOF data, and combining the extracted grass region masks for obstacle recognition.

[0015] To achieve the above and other related objectives, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store a computer program; the one or more processors are connected to the memories and are used to run the computer program to execute the lawnmower robot multi-sensor obstacle recognition method.

[0016] As described above, this invention provides a multi-sensor obstacle recognition method, system, and terminal for lawnmower robots, offering the following advantages: The lawnmower robot integrates a visual sensor and a time-of-flight (TOF) sensor. By acquiring images and TOF data, it processes the images and extracts a grass area mask. Then, based on the images and TOF data, it performs point cloud partitioning and grass height estimation, combining the grass area mask to achieve accurate obstacle recognition. This method combines the advantages of visual and TOF sensing technologies, constructing a more robust and efficient obstacle recognition system, significantly improving the lawnmower robot's adaptability in complex environments and ensuring its operational safety. This invention effectively reduces the limitations of lawnmower usage, significantly reduces missed and false detections of small obstacles due to grass obstruction, and avoids misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition. Attached Figure Description

[0017] Figure 1 The diagram shown is a flowchart illustrating a multi-sensor obstacle recognition method for a lawnmower robot according to an embodiment of the present invention.

[0018] Figure 2 The diagram shows a flowchart of extracting a mask for a grassland area according to an embodiment of the present invention.

[0019] Figure 3 The diagram shown is a flowchart illustrating obstacle recognition in one embodiment of the present invention.

[0020] Figure 4 The diagram shown is a structural schematic of a multi-sensor obstacle recognition system for a lawnmower robot according to an embodiment of the present invention.

[0021] Figure 5 The diagram shown is a structural schematic of an electronic terminal according to an embodiment of the present invention. Detailed Implementation

[0022] The following specific examples illustrate the implementation of the present invention. Those skilled in the art can easily understand other advantages and effects of the present invention from the content disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and various details in this specification can also be modified or changed based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that, unless otherwise specified, the following embodiments and features described therein can be combined with each other.

[0023] It should be noted that in the following description, reference is made to the accompanying drawings, which illustrate several embodiments of the invention. It should be understood that other embodiments may also be used, and changes in mechanical composition, structure, electrical system, and operation may be made without departing from the spirit and scope of the invention. The following detailed description should not be considered limiting, and the scope of the embodiments of the invention is defined only by the claims of the published patents. The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. Spatially related terms, such as “upper,” “lower,” “left,” “right,” “below,” “below,” “lower part,” “above,” “upper part,” etc., may be used herein to illustrate the relationship between one element or feature shown in the figures and another element or feature.

[0024] Throughout this specification, when it is said that a part is "connected" to another part, this includes not only "direct connection" but also "indirect connection" by placing other elements in between. Furthermore, when it is said that a part "includes" a certain constituent element, unless otherwise stated otherwise, this does not exclude other constituent elements, but rather means that other constituent elements may also be included.

[0025] The terms "first," "second," and "third," etc., used herein are for the purpose of describing various parts, components, regions, layers, and / or segments, but are not limiting. These terms are used only to distinguish one part, component, region, layer, or segment from others. Therefore, the "first part," "component," "region," "layer," or "segment" described below may refer to a "second part," "component," "region," "layer," or "segment" without departing from the scope of this invention.

[0026] Furthermore, as used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context indicates otherwise. It should be further understood that the terms “comprising,” “including,” indicate the presence of the stated feature, operation, element, component, item, kind, and / or group, but do not preclude the presence, occurrence, or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms “or” and “and / or” as used herein are interpreted as inclusive, or mean any one or any combination thereof. Thus, “A, B, or C” or “A, B, and / or C” means “any one of: A; B; C; A and B; A and C; B and C; A, B, and C.” Exceptions to this definition arise only when combinations of elements, functions, or operations are inherently mutually exclusive in some manner.

[0027] This invention provides a multi-sensor obstacle recognition method for lawnmower robots. It integrates a visual sensor and a time-of-flight (TOF) sensor into the lawnmower robot. By acquiring images and TOF data, the method processes the images and extracts a mask for the grass region. Then, based on the images and TOF data, it performs point cloud partitioning and grass height estimation, combining the grass region mask to achieve accurate obstacle recognition. This method combines the advantages of visual and TOF sensing technologies to construct a more robust and efficient obstacle recognition system, significantly improving the lawnmower robot's adaptability in complex environments and ensuring its operational safety. This invention effectively reduces the limitations of lawnmower usage, significantly reduces the missed and false detections of small obstacles caused by grass obstruction, and avoids the problem of misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition.

[0028] The present invention will now be described in detail with reference to the accompanying drawings, so that those skilled in the art can readily implement it. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0029] like Figure 1 This is a flowchart illustrating a multi-sensor obstacle recognition method for a lawnmower robot according to an embodiment of the present invention.

[0030] The method includes:

[0031] Step S1: Acquire images and TOF data collected by the vision sensor and time-of-flight ranging sensor installed on the lawnmower robot.

[0032] In one embodiment, a vision sensor and a time-of-flight ranging sensor are installed on the lawnmower robot.

[0033] The visual sensor can capture images of the surrounding environment in real time. This image data is presented in RGB (red, green, blue) color mode, providing the robot with rich visual information. RGB color mode is a color model that uses combinations of red, green, and blue to represent other colors. To further improve the accuracy and reliability of obstacle recognition, the visual sensor on the lawnmower robot can capture the location of environmental images, such as obstacles in front, to the sides, and even behind. Preferably, to improve processing efficiency, the image size is limited to 640x360 pixels or less.

[0034] A time-of-flight (TOF) ranging sensor is a device that measures the time it takes for light to travel from emission to reception. Through this measurement, a TOF sensor can calculate the distance between an object and the sensor, generating three-dimensional point cloud data. This point cloud data provides precise information about the object's position and shape in three-dimensional space, helping to identify the three-dimensional structure of obstacles.

[0035] Step S2: Process the image and extract the grass area mask.

[0036] In one embodiment, such as Figure 2 Step S2 includes:

[0037] The acquired images undergo distortion correction and image segmentation to obtain segmented images. Specifically, the acquired images are dedistorted to facilitate subsequent 3D point cloud projection. The distorted images are then subjected to region segmentation to obtain segmented images.

[0038] The segmented image is subjected to Gaussian filtering to obtain a filtered image. Specifically, Gaussian filtering is a linear smoothing filter used to remove Gaussian noise from the image while preserving its edge information. Gaussian filtering is applied to the segmented image to remove noise and improve image quality.

[0039] The filtered image is converted from the RGB color gamut to the HSV color gamut to obtain an HSV image; specifically, the filtered image is converted from the RGB color gamut to the HSV color gamut. The HSV color gamut (Hue, Saturation, Value) is a color model used to describe the characteristics of color. Hue represents the type of color, Saturation represents the vividness of the color, and Value represents the brightness of the color.

[0040] Based on the hue threshold obtained from the grassland test, the HSV image is segmented to obtain a preliminary grassland region mask; specifically, the HSV image is segmented according to the hue threshold obtained from the grassland test. The purpose of this step is to distinguish the grassland region from other regions in the image and obtain a preliminary grassland region mask.

[0041] The non-grass areas in the initial grassland area mask are subjected to erosion and dilation operations to obtain the processed mask. The erosion operation can remove small noise in the mask, while the dilation operation can smooth the area boundaries, making the mask more accurate.

[0042] The boundaries of non-grassland areas in the processed mask are extracted and their areas are calculated to extract the final grassland area mask. Specifically, by extracting the boundaries of non-grassland areas in the processed mask and calculating the areas of these areas, we can further filter out areas that do not belong to grassland and extract the final grassland area mask accordingly.

[0043] In one embodiment, the step of performing distortion processing and image region segmentation on the acquired image to obtain a segmented image includes:

[0044] Based on the calibrated image intrinsic parameters, distortion correction is performed on the image. Specifically, the camera needs to be calibrated to obtain its intrinsic parameters, including focal length, optical center coordinates, and distortion coefficients. Using these calibrated intrinsic parameters, distortion correction is applied to the original image. Distortions mainly include radial and tangential distortions, which can be eliminated through appropriate mathematical transformations (such as inverse transformations of barrel and pincushion distortions). The purpose of this step is to eliminate image distortion caused by camera lens distortions (such as barrel and pincushion distortions), thereby ensuring that the geometry in the image remains realistic and accurate. The distortion-corrected image will be easier to process in subsequent 3D point cloud projection and obstacle recognition.

[0045] The lower half of the image, after distortion correction, containing the grass region, is segmented as the segmented image. Specifically, considering that the grass region is typically located in the lower half of the image when the lawnmower robot is walking on grass, it is necessary to extract the lower half of the image as the segmented image. This step aims to reduce noise interference and algorithm processing time, as the image information in the upper half is usually not essential for grass region recognition. This invention, by extracting the lower half of the image as the segmented image, can help lawnmower robot systems reduce noise interference, lower algorithm processing time, and improve the efficiency of grass region recognition.

[0046] In one embodiment, the step of extracting the boundaries of non-grassland areas in the processed mask and calculating the area to extract the final grassland area mask includes:

[0047] The mask image is processed by extracting all closed boundaries of non-grassland areas to determine each sub-region, and then calculating the area of ​​the minimum bounding rectangle for each sub-region. Specifically, boundary detection algorithms (such as Canny edge detection, Sobel operator, Laplacian operator, etc.) are used to process the mask image to identify all closed boundaries of non-grassland areas. These boundaries segment the image into several independent sub-regions. For each sub-region, the area of ​​its minimum bounding rectangle is calculated.

[0048] Based on a set area threshold, small-area noise in the processed mask is filtered out according to the minimum bounding rectangle area of ​​each sub-region, resulting in the final grassland region mask. Specifically, a reasonable area threshold is set according to the application scenario and actual needs. This threshold is used to distinguish which non-grass areas are caused by noise or misidentification and which are truly non-grass areas that need attention. All sub-regions are traversed, and their minimum bounding rectangle areas are compared with the set area threshold. If the minimum bounding rectangle area of ​​a sub-region is less than the threshold, it is considered noise and filtered out from the mask. The purpose of this step is to remove small-area non-grass areas caused by image noise, misidentification, or processing errors.

[0049] Step S3: Based on the image and TOF data, perform point cloud partitioning and grass height estimation, and combine the extracted grass area mask to identify obstacles.

[0050] In one embodiment, such as Figure 3 Step S3 includes:

[0051] The acquired images are time-aligned with the Time-of-Flight (TOF) data; specifically, the acquired images and TOF data are time-synchronized. This ensures that the two are consistent in time, thus accurately reflecting the scene at the same moment. A timestamp interval of less than 100ms is recommended to reduce errors caused by time differences.

[0052] Point clouds in the aligned Time-of-Flight (TOF) data that can be projected onto the aligned image are designated as common-view point clouds, while point clouds that cannot be projected onto the aligned image are designated as non-common-view point clouds. Specifically, the TOF data point clouds are projected onto the aligned image, and based on how well they are projected, they are categorized as common-view point clouds and non-common-view point clouds. Common-view point clouds refer to those TOF data point clouds that can be projected onto the aligned image, meaning the corresponding area is jointly sensed by both the visual sensor and the time-of-flight ranging sensor. These point clouds establish a direct connection between 3D space and the 2D image. Non-common-view point clouds refer to those TOF data point clouds that cannot be successfully projected onto the aligned image, meaning the corresponding area is sensed by the time-of-flight ranging sensor but not by the visual sensor.

[0053] Voxel filtering downsampling is performed on the point cloud of the common view area to obtain the processed point cloud of the common view area; specifically, voxel filtering downsampling processing is performed on the point cloud of the common view area. This step aims to reduce the point cloud density and improve the efficiency of subsequent processing.

[0054] Calculate the adaptive grass height of the processed common view point cloud and determine the grass height range of the common view point cloud and the non-common view point cloud. Filter out the points in the common view point cloud and the non-common view point cloud that are within the corresponding grass height range.

[0055] Based on the grassland area mask, density clustering and obstacle identification of the filtered common view area point cloud are performed to obtain the obstacle point cloud of the common view area.

[0056] Density clustering and obstacle identification of point cloud clusters are performed on the filtered non-common view point cloud to obtain the obstacle point cloud of the non-common view.

[0057] The final obstacle point cloud is obtained by combining the obstacle point clouds from the common view area and the non-common view area. Specifically, the obstacle point clouds identified from the common view area and the non-common view area are merged to obtain the final obstacle point cloud.

[0058] In one embodiment, the step of calculating the adaptive grass height of the processed common-view point cloud and determining the grass height range of the common-view point cloud and non-common-view point clouds for point cloud filtering includes:

[0059] The height morphology distribution of the point cloud in the shared field of view is statistically processed, and the adaptive grass height of the shared field of view is calculated. Points in the processed shared field of view that are not higher than the adaptive grass height are then filtered out. Specifically, the height morphology distribution of the point cloud in the shared field of view is statistically processed, and combined with preset grass height morphology distribution features and obstacle height morphology distribution features, areas initially identified as grass are determined, and the adaptive grass height of the shared field of view is determined. Then, point clouds in the processed shared field of view that are not higher than the adaptive grass height are filtered out. The known grass height morphology distribution features and obstacle height morphology distribution features can be obtained based on prior knowledge (such as the geometry and physical characteristics of obstacles) or through learning algorithms (such as machine learning, deep learning, etc.).

[0060] The target grassland height in the non-common-view area is calculated based on the adaptive grassland height of the shared-view area and a preset threshold. Point clouds in the processed non-common-view area point cloud that are not higher than the target grassland height are then filtered out. Specifically, since the non-common-view area point cloud is connected to the shared-view area point cloud in the actual environment, the grassland height information of the shared-view area can be used to infer the grassland height in the non-common-view area to reduce computational load. The adaptive grassland height of the shared-view area is used as a benchmark, and then a preset threshold is added to obtain the target grassland height in the non-common-view area. This preset threshold is to ensure that the grassland point cloud can be cleanly filtered out in the non-common-view area. At the same time, considering factors such as terrain changes and vegetation height differences, a certain safety height margin is added to the filtering process. Then, point clouds in the processed non-common-view area point cloud that are not higher than the target grassland height are filtered out again.

[0061] In one embodiment, the step of performing density clustering and obstacle identification on the filtered common-view point cloud based on the grassland area mask to obtain the obstacle point cloud of the common-view area includes:

[0062] Density clustering is performed on the filtered common view area point cloud to form one or more point cloud clusters in the common view area;

[0063] The obstacle point cloud in the shared field of view is determined based on the position of the point cloud cluster projected onto the grass area mask and the height morphology distribution of the point cloud cluster. Specifically, each point cloud cluster is projected onto the grass area mask. If a point cloud cluster can be projected onto the mask, it is likely a grass area point cloud rather than an obstacle point cloud. For point cloud clusters that cannot be projected onto the grass area mask, their height morphology distribution is further analyzed. If the height morphology distribution of the statistically analyzed clustered point clouds matches the height morphology distribution characteristics of obstacles, it is determined to be an obstacle point cloud. This is achieved by: statistically analyzing the height of the point cloud clusters and comparing it with preset obstacle height distribution characteristics. If the height of the point cloud cluster matches the height range of obstacles, and its normal vector (representing the orientation of the point cloud surface) also matches the normal vector characteristics of obstacles (such as being perpendicular to the ground or having a certain tilt angle), then the point cloud cluster is determined to be an obstacle point cloud.

[0064] In one embodiment, the step of performing density clustering and obstacle determination on the filtered non-common-view point cloud to obtain the obstacle point cloud of the non-common-view area includes:

[0065] Density clustering is performed on the filtered non-common view point cloud to obtain non-common view point cloud clusters;

[0066] Obstacle point clouds in non-common-view areas are identified based on the morphological distribution of point cloud clusters. Specifically, in non-common-view areas, since image information cannot be directly used for identification, morphological distribution becomes an important criterion. First, the morphological distribution of point cloud clusters is statistically analyzed. If the morphological distribution of the statistically analyzed clusters of point clouds matches the morphological distribution characteristics of obstacles, then it is identified as an obstacle point cloud. The method includes: first, calculating the normal vector of each point in the point cloud cluster; if the calculated normal vector of the point cloud cluster matches the known normal vector of an obstacle, then the point cloud cluster is identified as an obstacle point cloud.

[0067] Similar to the above embodiments, the present invention provides a multi-sensor obstacle recognition system for lawn mowing robots.

[0068] The following specific embodiments are provided in conjunction with the accompanying drawings:

[0069] like Figure 4 This diagram illustrates the structure of a multi-sensor obstacle recognition system for a lawnmower robot according to an embodiment of the present invention.

[0070] The system includes:

[0071] Data acquisition module 1 is used to acquire images and TOF data collected by the vision sensor and time-of-flight ranging sensor installed on the lawnmower robot;

[0072] Grassland area mask extraction module 2, connected to the data acquisition module 1, is used to process the image and extract the grassland area mask;

[0073] The obstacle recognition module 3 is connected to the data acquisition module 1 and the grass area mask extraction module 2. It is used to perform point cloud partitioning and grass height estimation based on the image and TOF data, and to perform obstacle recognition in combination with the extracted grass area mask.

[0074] Since the implementation principle of the multi-sensor obstacle recognition system of the lawnmower robot has been described in the previous embodiments, it will not be repeated here.

[0075] The multi-sensor obstacle recognition method for lawnmower robots provided in this invention can be implemented on the terminal side or the server side. Regarding the hardware structure of the electronic terminal, please refer to [link to relevant documentation]. Figure 5This is a schematic diagram of an optional hardware structure of an electronic terminal 1000 provided in an embodiment of the present invention. The terminal 1000 can be a mobile phone, computer device, tablet device, personal digital processing device, factory back-end processing device, etc. The terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010, and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It is understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to a data bus, the bus system 1005 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 5 The general will label all buses as bus systems.

[0076] The user interface 1009 may include a monitor, keyboard, mouse, trackball, clicker, button, touchpad, or touch screen.

[0077] It is understood that memory 1002 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM) or programmable read-only memory (PROM), which serves as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM) and synchronous static random access memory (SSRAM). The memories described in the embodiments of this invention are intended to include, but are not limited to, these and any other suitable categories of memory.

[0078] In this embodiment of the invention, the memory 1002 is used to store various types of data to support the operation of the terminal 1000. Examples of this data include: any executable program for operation on the terminal 1000, such as the operating system 10021 and application program 10022; the operating system 10021 contains various system programs, such as the framework layer, core library layer, driver layer, etc., for implementing various basic services and handling hardware-based tasks. The application program 10022 may contain various applications, such as media players, browsers, etc., for implementing various application services. The implementation of the multi-sensor obstacle recognition method for lawnmower robots provided in this embodiment of the invention can be included in the application program 10022.

[0079] The methods disclosed in the above embodiments of the present invention can be applied to or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the integrated logic circuit of the hardware in the processor 1001 or by instructions in the form of software. The processor 1001 may be a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present invention. The processor 1001 may be a microprocessor or any conventional processor, etc. The steps of the accessory optimization method provided in the embodiments of the present invention can be directly reflected as being executed by a hardware decoding processor, or being executed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium, which is located in a memory. The processor reads the information in the memory and combines it with its hardware to complete the steps of the aforementioned method.

[0080] In an exemplary embodiment, the terminal 1000 may be used to execute the aforementioned method by one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), or complex programmable logic devices (CPLDs).

[0081] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented using computer program-related hardware. The aforementioned computer program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0082] In the embodiments provided in this application, the computer-readable and writable storage medium may include read-only memory, random access memory, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, flash memory, USB flash drive, portable hard drive, or any other medium capable of storing desired program code in the form of instructions or data structures and accessible by a computer. Additionally, any connection may be appropriately referred to as a computer-readable medium. For example, if instructions are transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are included in the definition of the medium. However, it should be understood that computer-readable and writable storage media and data storage media do not include connections, carrier waves, signals, or other transient media, but are intended for non-transient, tangible storage media. The disks and optical discs used in the application include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically copy data magnetically, while optical discs use lasers to copy data optically.

[0083] In summary, the multi-sensor obstacle recognition method, system, and terminal for lawnmower robots of this invention integrates a visual sensor and a time-of-flight (TOF) sensor into the lawnmower robot. By acquiring images and TOF data, the system processes the images and extracts a mask for the grass region. Then, based on the images and TOF data, it performs point cloud partitioning and grass height estimation, achieving accurate obstacle recognition by combining the grass region mask. This method combines the advantages of visual and TOF sensing technologies to construct a more robust and efficient obstacle recognition system, significantly improving the lawnmower robot's adaptability in complex environments and ensuring its operational safety. This invention effectively reduces the limitations of lawnmower usage, significantly reduces the missed and false detections of small obstacles caused by grass obstruction, and avoids the problem of misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition. Therefore, this invention effectively overcomes the various shortcomings of existing technologies and has high industrial application value.

[0084] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A multi-sensor obstacle recognition method for a lawnmower robot, characterized in that, The method includes: Acquire images and TOF data from the vision sensors and time-of-flight range sensors mounted on the lawnmower robot; The image is processed, and a mask for the grassy area is extracted; Based on the image and TOF data, point cloud partitioning and grass height estimation are performed, and obstacle recognition is performed by combining the extracted grass area mask. The step of performing point cloud partitioning and grass height estimation based on the image and TOF data, and then combining the extracted grass region mask for obstacle recognition, includes: temporally aligning the acquired image with the TOF data; identifying point clouds in the aligned TOF data that can be projected onto the aligned image as common-view point clouds and point clouds that cannot be projected onto the aligned image as non-common-view point clouds; performing voxel filtering downsampling on the common-view point clouds to obtain processed common-view point clouds; calculating the adaptive grass height of the processed common-view point clouds and determining the grass height range of the common-view point clouds and non-common-view point clouds for point cloud filtering; performing density clustering and point cloud cluster obstacle judgment on the filtered common-view point clouds based on the grass region mask to obtain obstacle point clouds in the common-view area; performing density clustering and point cloud cluster obstacle judgment on the filtered non-common-view point clouds to obtain obstacle point clouds in the non-common-view area; and combining the common-view and non-common-view obstacle point clouds to obtain the final obstacle point cloud.

2. The multi-sensor obstacle recognition method for lawnmower robots according to claim 1, characterized in that, The process of processing the image data and extracting the grass area mask includes: The acquired images are subjected to distortion processing and image region segmentation to obtain segmented images; The segmented image is subjected to Gaussian filtering to obtain a filtered image; The filtered image is converted from the RGB color gamut to the HSV color gamut to obtain an HSV image; Based on the hue threshold obtained from the grassland test, the HSV image is segmented to obtain a preliminary grassland region mask; The non-grassland areas in the initial grassland area mask are subjected to an erosion and expansion operation to obtain the treated mask. Extract the boundaries of non-grassland areas in the processed mask and calculate the area of ​​the area to extract the final grassland area mask.

3. The multi-sensor obstacle recognition method for lawnmower robots according to claim 2, characterized in that, The process of performing distortion processing and image region segmentation on the acquired image to obtain a segmented image includes: Based on the calibrated image intrinsic parameters, the image is subjected to distortion correction processing; The lower half of the image after distortion correction, containing the grassland region, is segmented as the segmented image.

4. The multi-sensor obstacle recognition method for lawnmower robots according to claim 2, characterized in that, The process of extracting the boundaries of non-grassland areas in the processed mask and calculating the area to extract the final grassland area mask includes: Extract all closed boundaries of non-grassland areas in the processed mask to determine each sub-region, and calculate the minimum bounding rectangle area of ​​each sub-region; Based on the set area threshold, small-area noise in the processed mask is filtered out according to the minimum bounding rectangle area of ​​each sub-region to obtain the final grassland area mask.

5. The multi-sensor obstacle recognition method for lawnmower robots according to claim 1, characterized in that, The process of calculating the adaptive grass height of the common-view point cloud and determining the grass height range of both the common-view point cloud and the non-common-view point cloud for point cloud filtering includes: The height morphology distribution of the common view area point cloud after statistical processing is calculated, and the adaptive grass height of the common view area is calculated. Point clouds in the processed common view area point cloud that are not higher than the adaptive grass height are filtered out. The target grass height in the non-common-view area is calculated based on the adaptive grass height of the common-view area and a preset threshold, and point clouds in the processed non-common-view area point cloud that are not higher than the target grass height are filtered out.

6. The multi-sensor obstacle recognition method for lawnmower robots according to claim 1, characterized in that, The process of performing density clustering and obstacle detection on the filtered common-view point cloud based on the grassland area mask to obtain the obstacle point cloud of the common-view area includes: Density clustering is performed on the filtered point cloud of the common view area to obtain one or more point cloud clusters of the common view area; The obstacle point cloud in the common field of view is determined based on the position of the point cloud cluster projected onto the mask of the grassland area and the height and shape distribution of the point cloud cluster.

7. The multi-sensor obstacle recognition method for lawnmower robots according to claim 1, characterized in that, The step of performing density clustering and obstacle identification on the filtered non-common view point cloud to obtain the obstacle point cloud of the non-common view includes: Density clustering is performed on the filtered non-common view point cloud to obtain non-common view point cloud clusters; Determine obstacle point clouds in non-common view areas based on the morphological distribution of point cloud clusters.

8. A multi-sensor obstacle recognition system for a lawnmower robot, characterized in that, The system includes: The data acquisition module is used to acquire images and TOF data collected by the vision sensors and time-of-flight ranging sensors installed on the lawnmower robot. A grassland area mask extraction module, connected to the data acquisition module, is used to process the image and extract the grassland area mask; An obstacle recognition module is connected to the data acquisition module and the grass area mask extraction module. It is used to perform point cloud partitioning and grass height estimation based on the image and TOF data, and to perform obstacle recognition in combination with the extracted grass area mask. The step of performing point cloud partitioning and grass height estimation based on the image and TOF data, and then combining the extracted grass region mask for obstacle recognition, includes: temporally aligning the acquired image with the TOF data; identifying point clouds in the aligned TOF data that can be projected onto the aligned image as common-view point clouds and point clouds that cannot be projected onto the aligned image as non-common-view point clouds; performing voxel filtering downsampling on the common-view point clouds to obtain processed common-view point clouds; calculating the adaptive grass height of the processed common-view point clouds and determining the grass height range of the common-view point clouds and non-common-view point clouds for point cloud filtering; performing density clustering and point cloud cluster obstacle judgment on the filtered common-view point clouds based on the grass region mask to obtain obstacle point clouds in the common-view area; performing density clustering and point cloud cluster obstacle judgment on the filtered non-common-view point clouds to obtain obstacle point clouds in the non-common-view area; and combining the common-view and non-common-view obstacle point clouds to obtain the final obstacle point cloud.

9. An electronic terminal, characterized in that, include: One or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors are connected to the memory and are used to run the computer program to perform the method as described in any one of claims 1 to 7.

Images