Multi-sensor obstacle identification method, system and terminal for mowing robot

Abstract

The invention provides a mowing robot multi-sensor obstacle recognition method and system and a terminal, a visual sensor and a flight time distance measuring sensor are integrated on a mowing robot, images and TOF data are collected, the images are processed, grassland area masks are extracted, point cloud partitioning and grassland height estimation are carried out based on the images and the TOF data, and the grassland height recognition accuracy is improved. And precise obstacle identification is realized in combination with a grassland area mask. According to the method, the advantages of the vision and TOF sensing technology are combined, a more robust and efficient obstacle recognition system is constructed, the adaptive capacity of the mowing robot in a complex environment is remarkably improved, and the working safety of the mowing robot is ensured. According to the invention, the use limitation of the mower is effectively reduced, the conditions of missing detection and false detection of small obstacles caused by the shielding of grassland are greatly reduced, and meanwhile, the problem that tall grass and weeds are wrongly recognized as obstacles is avoided, so that the accuracy of obstacle recognition is remarkably improved.

Description

Technical Field

[0001] The present invention relates to the technical field of lawn mowing robots, and in particular to a lawn mowing robot multi-sensor obstacle recognition method, system and terminal. Background Art

[0002] With the rapid development of intelligent technology, automated equipment is increasingly used in daily life. Especially in the fields of agriculture and gardening, automatic lawn mowing robots have gradually become an important tool for improving work efficiency and reducing labor costs. These robots are usually equipped with a variety of sensors to achieve autonomous navigation and environmental perception, so that they can work safely and efficiently in complex outdoor environments.

[0003] In lawn mowing robots, vision sensors and time-of-flight (TOF) sensors are two common sensor types. Vision sensors use cameras to capture images of the surrounding environment and perform obstacle recognition and path planning through image processing technology. The advantages of vision sensors are their high resolution and rich color information, which can effectively identify different types of obstacles, such as trees, stones, and other gardening facilities. However, the performance of vision sensors may be limited under extreme conditions such as strong light, shadows, and low light, and they cannot accurately obtain 3D information of obstacles, resulting in reduced recognition accuracy.

[0004] On the other hand, TOF sensors provide high-precision distance measurement by emitting laser pulses and measuring the return time to obtain depth information. The working principle of TOF sensors enables them to maintain good performance in environments with changing lighting conditions, and are suitable for identifying and locating obstacles. This makes TOF sensors play an important role in lawn mowing robots, especially in the identification of complex terrain and irregular obstacles. However, due to the lack of information such as color and texture, it is often impossible to determine whether the obstacles identified by TOF are accurate. Summary of the invention

[0005] In view of the shortcomings of the prior art mentioned above, the purpose of the present invention is to provide a multi-sensor obstacle recognition method, system and terminal for a lawn mowing robot, which are used to solve the technical problems in the prior art such as the large limitations on the use of the lawn mower when a single sensor is used for obstacle recognition, the missed detection and false detection of small obstacles due to grass occlusion, and the false identification of tall grass and weeds in the grass.

[0006] To achieve the above-mentioned object and other related objects, the present invention provides a multi-sensor obstacle recognition method for a lawn mowing robot, the method comprising: acquiring images and TOF data collected by a visual sensor and a time-of-flight ranging sensor provided on the lawn mowing robot; processing the images and extracting a grass area mask; performing point cloud partitioning and grass height estimation based on the images and TOF data, and performing obstacle recognition in combination with the extracted grass area mask.

[0007] In one embodiment of the present invention, the processing of the image data and the extraction of the grass area mask include: performing distortion processing and image area segmentation on the captured image to obtain a segmented image; performing Gaussian filtering on the segmented image to obtain a filtered image; converting the filtered image from the RGB color gamut to the HSV color gamut to obtain an HSV image; segmenting the HSV image based on the hue threshold obtained from the grass test to obtain a preliminary grass area mask; performing corrosion and expansion operations on the non-grass area in the preliminary grass area mask to obtain a processed mask; extracting the non-grass area boundary in the processed mask and calculating the area to extract the final grass area mask.

[0008] In one embodiment of the present invention, the distortion processing and image region segmentation of the captured image to obtain the segmented image includes: dedistorting the image based on the calibrated image internal parameters; and segmenting the lower half of the image after dedistortion where the grass area is located as the segmented image.

[0009] In one embodiment of the present invention, the extraction of non-grass area boundaries in the processed mask and calculation of the area area to extract the final grass area mask includes: extracting all closed boundaries of the non-grass area in the processed mask to determine each sub-region, and calculating the minimum outer enclosing rectangular area of ​​each sub-region; based on a set area threshold, filtering out small area noise in the processed mask according to the minimum outer enclosing rectangular area of ​​each sub-region to obtain the final grass area mask.

[0010] In one embodiment of the present invention, the point cloud partitioning and grass height estimation based on the image and TOF data, and obstacle identification in combination with the extracted grass area mask include: time aligning the captured image with the TOF data; taking the point cloud in the aligned TOF data that can be projected onto the aligned image as the common view area point cloud and taking the point cloud that cannot be projected onto the aligned image as the non-common view area point cloud; performing voxel filtering and downsampling on the common view area point cloud to obtain a processed common view area point cloud; calculating the adaptive grass height of the processed common view area point cloud and determining the grass height range of the common view area point cloud and the non-common view area point cloud to perform point cloud filtering; based on the grass area mask, performing density clustering and point cloud cluster obstacle judgment on the filtered common view area point cloud to obtain an obstacle point cloud of the common view area; performing density clustering and point cloud cluster obstacle judgment on the filtered non-common view area point cloud to obtain an obstacle point cloud of the non-common view area; combining the obstacle point clouds of the common view area and the non-common view area to obtain the final obstacle point cloud.

[0011] In one embodiment of the present invention, the calculation of the adaptive grass height of the common view area point cloud after processing and determining the grass height range of the common view area point cloud and the non-common view area for point cloud filtering include: statistically analyzing the height morphological distribution of the common view area point cloud after processing, calculating the adaptive grass height of the common view area, and filtering out the point clouds in the processed common view area point cloud that are not higher than the adaptive grass height; calculating the target grass height of the non-common view area based on the adaptive grass height of the common view area and a preset threshold, and filtering out the point clouds in the processed non-common view area point cloud that are not higher than the target grass height.

[0012] In one embodiment of the present invention, based on the grass area mask, density clustering and point cloud cluster obstacle judgment are performed on the filtered common view area point cloud to obtain the obstacle point cloud of the common view area, including: density clustering the filtered common view area point cloud to obtain one or more point cloud clusters of the common view area; judging the obstacle point cloud of the common view area based on the position of the point cloud cluster projected onto the grass area mask and the height morphological distribution of the point cloud cluster.

[0013] In one embodiment of the present invention, the density clustering of the filtered non-common view area point cloud and the point cloud cluster obstacle judgment to obtain the obstacle point cloud of the non-common view area includes: density clustering of the filtered non-common view area point cloud to obtain the point cloud cluster of the non-common view area; and judging the obstacle point cloud of the non-common view area based on the morphological distribution of the point cloud cluster.

[0014] To achieve the above-mentioned purpose and other related purposes, the present invention provides a multi-sensor obstacle recognition system for a lawn mowing robot, the system comprising: a data acquisition module, used to obtain images and TOF data collected by a visual sensor and a time-of-flight ranging sensor provided on the lawn mowing robot; a grass area mask extraction module, connected to the data acquisition module, used to process the image and extract the grass area mask; an obstacle recognition module, connected to the data acquisition module and the grass area mask extraction module, used to perform point cloud partitioning and grass height estimation based on the image and TOF data, and perform obstacle recognition in combination with the extracted grass area mask.

[0015] To achieve the above-mentioned purpose and other related purposes, the present invention provides an electronic terminal, comprising: one or more memories and one or more processors; the one or more memories are used to store computer programs; the one or more processors are connected to the memories and are used to run the computer program to execute the multi-sensor obstacle recognition method of the lawn mowing robot.

[0016] As described above, the present invention is a multi-sensor obstacle recognition method, system and terminal for a lawn mower robot, which has the following beneficial effects: the lawn mower robot of the present invention integrates a visual sensor and a time-of-flight ranging sensor, collects images and TOF data, processes the images and extracts the grass area mask, then performs point cloud partitioning and grass height estimation based on the images and TOF data, and realizes accurate obstacle recognition in combination with the grass area mask. This method combines the advantages of vision and TOF sensing technology to build a more robust and efficient obstacle recognition system, significantly improves the adaptability of the lawn mower robot in complex environments, and ensures its working safety. The present invention effectively reduces the limitations of the use of lawn mowers, greatly reduces the missed detection and false detection of small obstacles due to grass occlusion, and avoids the problem of misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 Shown is a flow chart of a multi-sensor obstacle recognition method for a lawn mowing robot in one embodiment of the present invention.

[0018] Figure 2 Shown is a schematic diagram of a process of extracting a grass area mask in an embodiment of the present invention.

[0019] Figure 3 Shown is a schematic diagram of the obstacle recognition process in one embodiment of the present invention.

[0020] Figure 4 Shown is a schematic structural diagram of a multi-sensor obstacle recognition system for a lawn mowing robot in one embodiment of the present invention.

[0021] Figure 5 Shown is a schematic structural diagram of an electronic terminal in an embodiment of the present invention. DETAILED DESCRIPTION

[0022] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the following embodiments and features in the embodiments can be combined with each other without conflict.

[0023] It should be noted that in the following description, reference is made to the accompanying drawings, which describe several embodiments of the present invention. It should be understood that other embodiments may also be used, and that mechanical composition, structural, electrical and operational changes may be made without departing from the spirit and scope of the present invention. The following detailed description should not be considered restrictive, and the scope of the embodiments of the present invention is limited only by the claims of the published patents. The terms used herein are only for describing specific embodiments and are not intended to limit the present invention. Spatially related terms, such as "upper", "lower", "left", "right", "below", "below", "lower", "above", "upper", etc., may be used in the text to facilitate the description of the relationship between an element or feature shown in the figure and another element or feature.

[0024] Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case of "direct connection" but also the case of "indirect connection" by placing other elements therebetween. In addition, when a part is said to "include" a certain constituent element, unless otherwise stated, it does not exclude other constituent elements, but means that other constituent elements may be included.

[0025] The terms first, second and third mentioned herein are used to describe various parts, components, regions, layers and / or segments, but are not limited thereto. These terms are only used to distinguish a certain part, component, region, layer or segment from other parts, components, regions, layers or segments. Therefore, the first part, component, region, layer or segment described below may refer to the second part, component, region, layer or segment within the scope of the present invention.

[0026] Furthermore, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless there is an indication to the contrary in the context. It should be further understood that the terms "comprise", "include" indicate the presence of the described features, operations, elements, components, items, kinds, and / or groups, but do not exclude the presence, occurrence or addition of one or more other features, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" used herein are interpreted as inclusive, or mean any one or any combination. Therefore, "A, B or C" or "A, B and / or C" means "any of the following: A; B; C; A and B; A and C; B and C; A, B and C". Exceptions to this definition will only occur when the combination of elements, functions or operations is inherently mutually exclusive in some way.

[0027] The present invention provides a multi-sensor obstacle recognition method for a lawn mower robot, which integrates a visual sensor and a time-of-flight ranging sensor in the lawn mower robot, collects images and TOF data, processes the images and extracts a grass area mask, then performs point cloud partitioning and grass height estimation based on the images and TOF data, and realizes accurate obstacle recognition in combination with the grass area mask. The method combines the advantages of vision and TOF sensing technologies to build a more robust and efficient obstacle recognition system, significantly improves the adaptability of the lawn mower robot in complex environments, and ensures its working safety. The present invention effectively reduces the limitations of the use of lawn mowers, greatly reduces the missed detection and false detection of small obstacles due to grass occlusion, and avoids the problem of misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition.

[0028] The following is a detailed description of the embodiments of the present invention with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention. The present invention can be embodied in many different forms and is not limited to the embodiments described herein.

[0029] like Figure 1 A schematic flow chart showing a multi-sensor obstacle recognition method for a lawn mowing robot in an embodiment of the present invention.

[0030] The method comprises:

[0031] Step S1: Acquire images and TOF data collected by a visual sensor and a time-of-flight ranging sensor provided on the lawn mowing robot.

[0032] In one embodiment, a visual sensor and a time-of-flight ranging sensor are provided on the lawn mowing robot.

[0033] The visual sensor can capture images of the surrounding environment in real time. These image data are presented in RGB (red, green, and blue) color mode, providing the robot with rich visual information. The RGB color mode is a color model that represents other colors based on the combination of red, green, and blue. In order to further improve the accuracy and reliability of obstacle recognition, the visual sensor can capture the location of the environmental image on the mowing robot, for example, it can capture obstacle information including the front, side, and even the rear. Preferably, in order to improve processing efficiency, the image size is limited to 640x360 pixels.

[0034] A time-of-flight sensor, also known as a TOF sensor, is a device that measures the time it takes for light to be emitted and received. Through this measurement, the TOF sensor can calculate the distance between the object and the sensor and generate three-dimensional point cloud data. These point cloud data provide the precise position and shape information of the object in three-dimensional space, which helps to identify the three-dimensional structure of obstacles.

[0035] Step S2: Process the image and extract the grass area mask.

[0036] In one embodiment, if Figure 2 , step S2 comprises:

[0037] The collected image is subjected to distortion processing and image segmentation to obtain a segmented image; specifically, the collected image is subjected to dedistortion processing to facilitate subsequent 3D point cloud projection. The distorted image is subjected to regional segmentation to obtain a segmented image.

[0038] The segmented image is subjected to Gaussian filtering to obtain a filtered image; specifically, Gaussian filtering is a linear smoothing filter used to remove Gaussian noise in an image while retaining edge information of the image. The segmented image is subjected to Gaussian filtering to remove noise points in the image and improve image quality.

[0039] The filtered image is converted from the RGB color gamut to the HSV color gamut to obtain an HSV image; specifically, the filtered image is converted from the RGB color gamut to the HSV color gamut. The HSV color gamut (Hue, Saturation, Value) is a color model used to describe the characteristics of color, where the hue (Hue) indicates the type of color, the saturation (Saturation) indicates the vividness of the color, and the value (Value) indicates the brightness of the color.

[0040] Based on the hue threshold obtained from the grass test, the HSV image is segmented to obtain a preliminary grass area mask; specifically, the HSV image is segmented based on the hue threshold obtained from the grass test. The purpose of this step is to distinguish the grass area from other areas in the image and obtain a preliminary grass area mask.

[0041] An erosion and dilation operation is performed on the non-grass area in the preliminary grass area mask to obtain a processed mask; the erosion operation can remove small noise points in the mask, while the dilation operation can smooth the area boundary to make the mask more accurate.

[0042] The non-grass area boundaries in the processed mask are extracted and the area of ​​the area is calculated to extract the final grass area mask. Specifically, by extracting the non-grass area boundaries in the processed mask and calculating the area of ​​these areas, we can further filter out areas that do not belong to the grass, and extract the final grass area mask accordingly.

[0043] In one embodiment, the performing distortion processing and image region segmentation on the acquired image to obtain the segmented image includes:

[0044] Based on the calibrated image internal parameters, the image is dedistorted; specifically, the camera needs to be calibrated to obtain the camera's internal parameters (internal parameters) including focal length, optical center coordinates, distortion coefficients, etc. The original image is dedistorted using the internal parameters obtained by calibration. Distortions mainly include radial distortion and tangential distortion, which can be eliminated through appropriate mathematical transformations (such as the inverse transformation of barrel distortion and pincushion distortion). The purpose of this step is to eliminate image distortion caused by camera lens distortion (such as barrel distortion, pincushion distortion, etc.), thereby ensuring that the geometric shapes in the image remain true and accurate. The dedistorted image will be easier for subsequent 3D point cloud projection and obstacle recognition.

[0045] The lower half of the image after dedistortion processing where the grass area is segmented is used as the segmented image. Specifically, considering that when the mowing robot walks on the grass, the grass area is usually located in the lower half of the image. Therefore, it is necessary to extract the lower half of the image as the segmented image. This step is intended to reduce noise interference and time-consuming algorithm processing, because the image information of the upper half is usually not necessary for the recognition of the grass area. The present invention extracts the lower half of the image as a segmented image, which can help the mowing robot system reduce noise interference, reduce algorithm processing time, and improve the recognition efficiency of the grass area.

[0046] In one embodiment, the extracting the non-grass region boundary in the processed mask and calculating the region area to extract the final grass region mask includes:

[0047] Extract all closed boundaries of the non-grass area in the processed mask to determine each sub-region, and calculate the minimum outer bounding rectangle area of ​​each sub-region; specifically, use a boundary detection algorithm (such as Canny edge detection, Sobel operator, Laplacian operator, etc.) to process the mask image to identify all closed boundaries of the non-grass area. These boundaries divide the image into several independent sub-regions. For each sub-region, calculate the area of ​​its minimum outer bounding rectangle.

[0048] Based on the set area threshold, the small area noise in the processed mask is filtered out according to the minimum outer enclosing rectangle area of ​​each sub-region to obtain the final grass area mask. Specifically, a reasonable area threshold is set according to the application scenario and actual needs. This threshold is used to distinguish which non-grass areas are caused by noise or misidentification and which are the non-grass areas that really need attention. Traverse all sub-regions and compare their minimum outer enclosing rectangle area with the set area threshold. If the minimum outer enclosing rectangle area of ​​a sub-region is less than the threshold, it is regarded as noise and filtered out from the mask. The purpose of this step is to remove small non-grass areas caused by image noise, misidentification or processing errors.

[0049] Step S3: performing point cloud partitioning and grass height estimation based on the image and TOF data, and performing obstacle recognition in combination with the extracted grass area mask.

[0050] In one embodiment, if Figure 3 , step S3 comprises:

[0051] Time-align the captured image with the TOF data; specifically, time-synchronize the captured image with the TOF data. This is to ensure that the two are consistent in time, so that they can accurately reflect the scene at the same moment. The timestamp interval is recommended to be less than 100ms to reduce the error caused by time difference.

[0052] The point cloud in the aligned TOF data that can be projected onto the aligned image is taken as the common view area point cloud, and the point cloud that cannot be projected onto the aligned image is taken as the non-common view area point cloud; specifically, the TOF data point cloud is projected onto the aligned image, and is divided into the common view area point cloud and the non-common view area point cloud according to the situation of the TOF data point cloud being projected onto the aligned image. The common view area point cloud refers to those TOF data point clouds that can be projected onto the aligned image, that is, the corresponding area is the area that can be sensed by the visual sensor and the time-of-flight ranging sensor. These point clouds establish a direct connection between the three-dimensional space and the two-dimensional image. The non-common view area point cloud refers to those TOF data point clouds that cannot be successfully projected onto the aligned image, that is, the corresponding area is the area that can be sensed by the time-of-flight ranging sensor, but not by the visual sensor.

[0053] The point cloud of the common view area is subjected to voxel filtering and downsampling to obtain a processed point cloud of the common view area; specifically, the point cloud of the common view area is subjected to voxel filtering and downsampling processing. This step aims to reduce the density of the point cloud and improve the efficiency of subsequent processing.

[0054] Calculate the adaptive grass height of the processed common view area point cloud and determine the grass height range of the common view area point cloud and the non-common view area point cloud, and filter out the point clouds within the corresponding grass height range in the common view area point cloud and the non-common view area point cloud;

[0055] Based on the grass area mask, density clustering and point cloud cluster obstacle judgment are performed on the filtered common view area point cloud to obtain the obstacle point cloud of the common view area;

[0056] Density clustering and point cloud cluster obstacle judgment are performed on the filtered non-common view area point cloud to obtain the obstacle point cloud of the non-common view area;

[0057] The obstacle point clouds in the common view area and the non-common view area are combined to obtain the final obstacle point cloud. Specifically, the obstacle point clouds identified in the common view area and the non-common view area are merged to obtain the final obstacle point cloud.

[0058] In one embodiment, the step of calculating the adaptive grass height of the processed common view area point cloud and determining the grass height range of the common view area point cloud and the non-common view area point cloud to perform point cloud filtering includes:

[0059] The height morphological distribution of the point cloud in the common view area after statistical processing is calculated, and the adaptive grass height in the common view area is calculated, and the point cloud in the processed common view area point cloud that is not higher than the adaptive grass height is filtered out; specifically, the height morphological distribution of the point cloud in the common view area after statistical processing is calculated, and the area preliminarily judged as grass is determined in combination with the preset grass height morphological distribution characteristics and the obstacle height morphological distribution characteristics, and the adaptive grass height in the common view area is determined; then, the point cloud in the processed common view area point cloud that is not higher than the adaptive grass height is filtered out. The known grass height morphological distribution characteristics and obstacle height morphological distribution characteristics can be obtained based on prior knowledge (such as the geometric shape and physical characteristics of the obstacle, etc.) or through learning algorithms (such as machine learning, deep learning, etc.).

[0060] The target grass height of the non-common view area is calculated based on the adaptive grass height of the common view area and the preset threshold, and the point cloud in the processed non-common view area point cloud that is not higher than the target grass height is filtered out; specifically, since the non-common view area point cloud is connected to the common view area point cloud in the actual environment, in order to reduce the amount of calculation, the grass height information of the common view area can be used to infer the grass height of the non-common view area. The adaptive grass height of the common view area is used as a benchmark, and then a preset threshold is added to obtain the target grass height of the non-common view area. This preset threshold is to ensure that the grass point cloud can be cleanly filtered out in the non-common view area. At the same time, taking into account factors such as terrain changes and vegetation height differences, a certain safety height space is added to the filtering process. Then, the point cloud in the processed non-common view area point cloud that is not higher than the target grass height is filtered out.

[0061] In one embodiment, based on the grass area mask, performing density clustering and point cloud cluster obstacle judgment on the filtered common view area point cloud to obtain the obstacle point cloud of the common view area includes:

[0062] Density clustering is performed on the filtered common view area point cloud to form one or more point cloud clusters in the common view area;

[0063] The obstacle point cloud in the common view area is determined based on the position of the point cloud cluster projected onto the grass area mask and the height morphological distribution of the point cloud cluster. Specifically, each point cloud cluster is projected onto the grass area mask. If the point cloud cluster can be projected onto the mask, then the point cloud cluster is likely to be a grass area point cloud rather than an obstacle point cloud. For point cloud clusters that cannot be projected onto the grass area mask, their height morphological distribution is further analyzed. If the statistical height morphological distribution of the clustered cluster point cloud meets the obstacle height morphological distribution characteristics, it is determined to be an obstacle point cloud. The method includes: counting the height of the point cloud cluster and comparing it with the preset obstacle height distribution characteristics. If the height of the point cloud cluster meets the height range of the obstacle, and its normal vector (indicating the orientation of the point cloud surface) also meets the normal vector characteristics of the obstacle (such as perpendicular to the ground or with a certain inclination angle), the point cloud cluster is determined to be an obstacle point cloud.

[0064] In one embodiment, performing density clustering and point cloud cluster obstacle judgment on the filtered non-common view area point cloud to obtain the obstacle point cloud of the non-common view area includes:

[0065] Density clustering is performed on the filtered non-common view area point cloud to obtain the point cloud cluster of the non-common view area;

[0066] The obstacle point cloud in the non-common view area is judged based on the morphological distribution of the point cloud cluster; specifically, in the non-common view area, since it is impossible to directly use image information for judgment, the morphological distribution becomes an important basis for judgment. First, the morphological distribution of the point cloud cluster is counted. If the morphological distribution of the clustered cluster point cloud meets the characteristics of the obstacle morphological distribution, it is judged as an obstacle point cloud. The method includes: first, the normal vector of each point cloud cluster is counted. If the calculated point cloud cluster normal vector meets the known obstacle normal vector, the point cloud cluster is judged to be an obstacle point cloud.

[0067] Similar to the principle of the above embodiment, the present invention provides a multi-sensor obstacle recognition system for a lawn mowing robot.

[0068] The following provides specific embodiments in conjunction with the accompanying drawings:

[0069] like Figure 4 A structural schematic diagram of a multi-sensor obstacle recognition system for a lawn mowing robot in an embodiment of the present invention is shown.

[0070] The system comprises:

[0071] The data acquisition module 1 is used to acquire images and TOF data collected by the visual sensor and the time-of-flight ranging sensor provided on the lawn mowing robot;

[0072] A grass area mask extraction module 2, connected to the data acquisition module 1, for processing the image and extracting a grass area mask;

[0073] The obstacle recognition module 3 is connected to the data acquisition module 1 and the grass area mask extraction module 2, and is used to perform point cloud partitioning and grass height estimation based on the image and TOF data, and to perform obstacle recognition in combination with the extracted grass area mask.

[0074] Since the implementation principle of the multi-sensor obstacle recognition system of the lawn mowing robot has been described in the above-mentioned embodiment, it will not be repeated here.

[0075] The multi-sensor obstacle recognition method for a lawn mowing robot provided in the embodiment of the present invention can be implemented on the terminal side or the server side. As for the hardware structure of the electronic terminal, please refer to Figure 5, is an optional hardware structure diagram of an electronic terminal 1000 provided in an embodiment of the present invention. The terminal 1000 may be a mobile phone, a computer device, a tablet device, a personal digital processing device, a factory background processing device, etc. The terminal 1000 includes: at least one processor 1001, a memory 1002, at least one network interface 10010 and a user interface 1009. The various components in the device are coupled together through a bus system 1005. It can be understood that the bus system 1005 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 1005 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, in Figure 5 In the specification, various buses are labeled as bus systems.

[0076] The user interface 1009 may include a display, a keyboard, a mouse, a trackball, a click gun, keys, buttons, a touch pad or a touch screen.

[0077] It is understood that the memory 1002 can be a volatile memory or a non-volatile memory, and can also include both volatile and non-volatile memories. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), which is used as an external cache. By way of exemplary but not limiting explanation, many forms of RAM are available, such as static random access memory (SRAM), synchronous static random access memory (SSRAM). The memory described in the embodiments of the present invention is intended to include but is not limited to these and any other suitable categories of memory.

[0078] The memory 1002 in the embodiment of the present invention is used to store various categories of data to support the operation of the terminal 1000. Examples of these data include: any executable program for operating on the terminal 1000, such as an operating system 10021 and an application 10022; the operating system 10021 includes various system programs, such as a framework layer, a core library layer, a driver layer, etc., for implementing various basic services and processing hardware-based tasks. The application 10022 may include various applications, such as a media player (MediaPlayer), a browser (Browser), etc., for implementing various application services. The implementation of the multi-sensor obstacle recognition method of the mowing robot provided in the embodiment of the present invention may be included in the application 10022.

[0079] The method disclosed in the above embodiment of the present invention can be applied to the processor 1001, or implemented by the processor 1001. The processor 1001 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit in the processor 1001 or the instruction in the form of software. The above processor 1001 may be a general processor, a digital signal processor (DSP, Digital Signal Processor), or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The processor 1001 can implement or execute the methods, steps and logic block diagrams disclosed in the embodiment of the present invention. The processor 1001 may be a microprocessor or any conventional processor, etc. In combination with the steps of the accessory optimization method provided in the embodiment of the present invention, it can be directly embodied as a hardware decoding processor to execute, or it can be executed by a combination of hardware and software modules in the decoding processor. The software module can be located in a storage medium, which is located in a memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0080] In an exemplary embodiment, the terminal 1000 may be implemented by one or more application specific integrated circuits (ASIC), DSP, programmable logic device (PLD), complex programmable logic device (CPLD) to execute the aforementioned method.

[0081] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to the computer program. The aforementioned computer program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, magnetic disk or optical disk and other media that can store program codes.

[0082] In the embodiments provided in the present application, the computer readable and writable storage medium may include a read-only memory, a random access memory, an EEPROM, a CD-ROM or other optical disk storage device, a disk storage device or other magnetic storage device, a flash memory, a USB flash drive, a mobile hard disk, or any other medium that can be used to store a desired program code in the form of an instruction or data structure and can be accessed by a computer. In addition, any connection can be appropriately referred to as a computer-readable medium. For example, if the instruction is sent from a website, a server or other remote source using a coaxial cable, an optical fiber cable, a twisted pair, a digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwaves, the coaxial cable, optical fiber cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwaves are included in the definition of the medium. However, it should be understood that computer readable and writable storage media and data storage media do not include connections, carriers, signals, or other temporary media, but are intended to be non-temporary, tangible storage media. Disk and disc, as used in this application, includes compact disc (CD), laser disc, optical disc, digital versatile disc (DVD), floppy disk and Blu-ray disc where disks usually reproduce data magnetically, while discs reproduce data optically with lasers.

[0083] In summary, the multi-sensor obstacle recognition method, system and terminal of the lawn mower robot of the present invention integrates a visual sensor and a time-of-flight ranging sensor in the lawn mower robot, collects images and TOF data, processes the images and extracts the grass area mask, and then performs point cloud partitioning and grass height estimation based on the images and TOF data, and realizes accurate obstacle recognition in combination with the grass area mask. This method combines the advantages of vision and TOF sensing technology to build a more robust and efficient obstacle recognition system, significantly improves the adaptability of the lawn mower robot in complex environments, and ensures its working safety. The present invention effectively reduces the limitations of the use of lawn mowers, greatly reduces the missed detection and false detection of small obstacles caused by grass occlusion, and avoids the problem of misidentifying tall grass and weeds as obstacles, thereby significantly improving the accuracy of obstacle recognition. Therefore, the present invention effectively overcomes the various shortcomings of the prior art and has a high industrial utilization value.

[0084] The above embodiments are merely illustrative of the principles and effects of the present invention, and are not intended to limit the present invention. Anyone familiar with the technology may modify or change the above embodiments without violating the spirit and scope of the present invention. Therefore, all equivalent modifications or changes made by a person of ordinary skill in the art without departing from the spirit and technical ideas disclosed by the present invention shall still be covered by the claims of the present invention.

Claims

1. A multi-sensor obstacle recognition method for a lawn mowing robot, characterized in that: The method comprises: Acquire images and TOF data collected by a visual sensor and a time-of-flight ranging sensor provided on the lawn mowing robot; Processing the image and extracting a grass area mask; Point cloud partitioning and grass height estimation are performed based on the image and TOF data, and obstacle recognition is performed in combination with the extracted grass area mask.

2. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 1, characterized in that: The step of processing the image data and extracting a grass area mask comprises: Perform distortion processing and image region segmentation on the acquired image to obtain a segmented image; Performing Gaussian filtering on the segmented image to obtain a filtered image; Convert the filtered image from the RGB color domain to the HSV color domain to obtain an HSV image; Based on the hue threshold obtained from the grass test, the HSV image is segmented to obtain a preliminary grass area mask; Performing an erosion and dilation operation on the non-grass area in the preliminary grass area mask to obtain a processed mask; The non-grass area boundary in the processed mask is extracted and the area of ​​the region is calculated to extract the final grass area mask.

3. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 2, characterized in that: The step of performing distortion processing and image region segmentation on the acquired image to obtain a segmented image comprises: Based on the calibrated image internal parameters, dedistorting the image; The lower half of the dedistorted image where the grass area is segmented is used as the segmented image.

4. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 2, characterized in that: The step of extracting the non-grass area boundary in the processed mask and calculating the area of ​​the region to extract the final grass area mask includes: Extract all closed boundaries of the non-grass area in the processed mask to determine each sub-area, and calculate the minimum outer bounding rectangle area of ​​each sub-area; Based on the set area threshold, small area noise in the processed mask is filtered out according to the minimum outer bounding rectangle area of ​​each sub-region to obtain the final grassland area mask.

5. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 1, characterized in that: The performing point cloud partitioning and grass height estimation based on the image and TOF data, and performing obstacle recognition in combination with the extracted grass area mask includes: Time-align the acquired images with the TOF data; The point cloud in the aligned TOF data that can be projected onto the aligned image is taken as the common view area point cloud, and the point cloud that cannot be projected onto the aligned image is taken as the non-common view area point cloud; Perform voxel filtering and downsampling on the common view area point cloud to obtain a processed common view area point cloud; Calculating the adaptive grass height of the processed common view area point cloud and determining the grass height range of the common view area point cloud and the non-common view area to filter out the point cloud; Based on the grass area mask, density clustering and point cloud cluster obstacle judgment are performed on the filtered common view area point cloud to obtain the obstacle point cloud of the common view area; Density clustering and point cloud cluster obstacle judgment are performed on the filtered non-common view area point cloud to obtain the obstacle point cloud of the non-common view area; The obstacle point cloud in the common view area and the non-common view area is combined to obtain the final obstacle point cloud.

6. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 5, characterized in that: The step of calculating the adaptive grass height of the processed common view area point cloud and determining the grass height range of the common view area point cloud and the non-common view area point cloud to filter out the point cloud includes: Statistically analyzing the height morphological distribution of the processed common view area point cloud, calculating the height of the adaptive grassland in the common view area, and filtering out the point cloud that is not higher than the adaptive grassland height in the processed common view area point cloud; The target grass height of the non-common view area is calculated based on the adaptive grass height of the common view area and a preset threshold, and the point cloud that is not higher than the target grass height in the processed non-common view area point cloud is filtered out.

7. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 5, characterized in that: The step of performing density clustering and point cloud cluster obstacle judgment on the filtered common view area point cloud based on the grass area mask to obtain the obstacle point cloud of the common view area includes: Density clustering is performed on the filtered common view area point cloud to obtain one or more point cloud clusters in the common view area; The obstacle point cloud in the common view area is determined based on the position of the point cloud cluster projected onto the grass area mask and the height morphological distribution of the point cloud cluster.

8. The multi-sensor obstacle recognition method for a lawn mowing robot according to claim 5, characterized in that: The step of performing density clustering and point cloud cluster obstacle judgment on the filtered non-common view area point cloud to obtain the obstacle point cloud of the non-common view area includes: Density clustering is performed on the filtered non-common view area point cloud to obtain the point cloud cluster of the non-common view area; Determine obstacle point clouds in non-common view areas based on the morphological distribution of point cloud clusters.

9. A multi-sensor obstacle recognition system for a lawn mowing robot, characterized in that: The system comprises: A data acquisition module, used to acquire images and TOF data collected by a visual sensor and a time-of-flight ranging sensor provided on the mowing robot; A grass area mask extraction module, connected to the data acquisition module, for processing the image and extracting a grass area mask; The obstacle recognition module is connected to the data acquisition module and the grass area mask extraction module, and is used to perform point cloud partitioning and grass height estimation based on the image and TOF data, and to perform obstacle recognition in combination with the extracted grass area mask.

10. An electronic terminal, characterized in that: include: one or more memories and one or more processors; The one or more memories are used to store computer programs; The one or more processors, connected to the memory, are configured to run the computer program to perform the method according to any one of claims 1 to 8.

Images