Lawn trimming robot environment sensing method and device based on fusion of vision and millimeter wave radar

Abstract

The invention belongs to the field of multi-sensor fusion, and provides a lawn trimming robot environment sensing method and device based on vision and millimeter wave radar fusion. The method comprises the following steps: constructing a lawn data set, optimizing an activation function of a YOLOv11 instance segmentation model in a training stage, simultaneously adjusting and optimizing a model hyper-parameter and an optimizer to inhibit overfitting, and deploying the model to an AI chip end after conversion and quantification, so as to realize real-time visual reasoning of camera data; a DBSCAN algorithm is introduced at a millimeter wave radar end to cluster original point clouds so as to extract stable clustering center points, and the influence of stray points on the detection precision is significantly reduced; the system completes time synchronization of cross-sensor data by using a software timestamp in combination with linear interpolation, realizes spatial alignment through coordinate transformation, then projects a radar clustering center point to a visual detection plane, and performs position matching and decision-level information fusion through a geometrical relationship between the center point and a detection frame, so as to obtain a detection result. And the operation overhead is remarkably reduced while high-precision target positioning is ensured.

Description

Technical Field

[0001] This invention relates to the field of multi-sensor fusion, specifically to an environmental perception method and apparatus for lawn mowing robots based on the fusion of vision and millimeter-wave radar. Background Technology

[0002] With the rapid development of artificial intelligence and sensing technologies, intelligent lawnmowers, as key equipment in the fields of landscaping and agricultural automation, are gradually replacing traditional artificial lawn mowing methods. Currently, there are three main environmental perception solutions for lawnmowers: pure vision, RTK + vision, and LiDAR + vision. The pure vision solution heavily relies on complex visual algorithms and is susceptible to interference from factors such as lighting and shadows, leading to false or missed obstacle detections, resulting in blade wear or even breakage. The RTK + vision solution relies heavily on satellite visibility, and in densely wooded or building-covered areas, it is easily affected by signal blockage and multipath effects, leading to positioning failures or significant accuracy reductions. Furthermore, the visual camera struggles to effectively detect obstacles obscured by lawns or other objects. The LiDAR + vision solution has high system cost requirements, and when working on open lawns or surfaces with specific textures, it may encounter false point clouds ("illusory obstacles"), causing the lawnmower to stop or detour unnecessarily, significantly reducing work efficiency. Summary of the Invention

[0003] The purpose of this invention is to propose an environmental perception method and device for lawn mowing robots based on the fusion of vision and millimeter-wave radar, so as to solve one or more technical problems existing in the prior art, and at least provide a beneficial option or create conditions.

[0004] This invention utilizes a visual sensor, a millimeter-wave radar sensor, an AI chip, and a machine vision processing system. The visual sensor and the millimeter-wave radar sensor acquire image data and location information, respectively, and the AI ​​chip and the machine vision processing system process the acquired image data and location information. The method of this invention first constructs a lawn dataset, then trains the dataset using a YOLOv11 instance segmentation model optimized with activation functions, and modifies the model hyperparameters and optimizer to prevent overfitting. Finally, after model transformation and quantization, it is deployed to an AI chip to complete visual inference. At the millimeter-wave radar end, the DBSCAN point cloud clustering algorithm is used to abstract the cluttered point cloud for object detection into cluster center points, obtaining relevant location information of the detected targets. Regarding sensor fusion, time alignment of the two sensors is achieved through software timestamps and linear interpolation, and spatial alignment is achieved through spatial position transformation. Finally, the radar cluster center points are mapped to the visual detection plane, and association determination is achieved through spatial matching of the target position and the visual detection box. The method specifically includes the following steps: Step 1: Construct a lawn dataset, adjust the relevant parameters of the YOLOv11 instance segmentation model, complete the training and quantization deployment of the YOLOv11 instance segmentation model, and realize the preprocessing of visual sensor data; Step 2: Analyze the millimeter-wave radar point cloud data and use the DBSCAN algorithm to perform clustering processing of the point cloud, thereby realizing the preprocessing of the millimeter-wave radar data; Step 3: Use software timestamps and linear interpolation to achieve time alignment of the two sensors, and complete spatial alignment of the two sensors through spatial position transformation relationship; Step 4: Project the cluster center points acquired by the millimeter-wave radar onto the camera's visual detection plane, and fuse the decision information of the two sensors based on the relevant positional relationship between the cluster center points and the detection boxes, and finally output the fusion result.

[0005] Further, the feature is that, in step one, images of lawns, boundaries, and obstacles are acquired using a visual sensor, and some lawn data is collected from the Roboflow website. The images acquired by the visual sensor and the collected lawn data are finely annotated to construct a lawn dataset. The lawn dataset is then trained using the YOLOv11 instance segmentation model. Considering the limited computing resources of embedded chips, the relevant parameters and optimizer of the model are modified before training the lawn dataset using the YOLOv11 instance segmentation model. The modifications are as follows: the activation function is changed from SiLU to ReLU; the model parameter lr0 is changed from 0.01 to 0.001, and lrf is changed from 0.01 to 0.9; the optimizer is changed from SGD to AdamW to prevent model overfitting. The parameter lr0 is the initial learning rate, and the parameter lrf is the learning rate factor.

[0006] Furthermore, in step two, a radar parsing program is written according to the millimeter-wave radar communication protocol to obtain the point cloud information of the target. The target is the point cloud information of obstacles in the lawn obtained by the millimeter-wave radar. The DBSCAN point cloud clustering algorithm is used to complete the clustering processing of the point cloud information of the target, converting the scattered point cloud of the detected target into cluster center points, and using the information of the cluster center points to represent the relevant location information of the detected target.

[0007] Furthermore, in step three, software timestamps are used to synchronize visual sensor data and millimeter-wave radar sensor data. Detection messages from both sensors are obtained through a RabbitMQ message queue. Then, the detection messages from the two sensors are time-aligned based on a timestamp interpolation method. A program is written to obtain the timestamps of the millimeter-wave radar and the visual sensor, and calculate the time difference between them. Then, using the radar data timestamp as a reference, linear interpolation is used to time-align the camera data, ensuring that the timestamps of the camera data and the radar data are consistent, thus obtaining the synchronized data. The linear interpolation function is as follows: ; , Given data from two known points, Let x be the x-coordinate of the point to be interpolated. for The interpolation function value is obtained at the given location.

[0008] Furthermore, in step four, the target's position in the lawn is first detected by millimeter-wave radar, and the target position is transformed into the millimeter-wave radar coordinate system. Then, the millimeter-wave radar coordinate system is transformed into the world coordinate system through coordinate transformation relationships. Next, the target data in the world coordinate system is transformed into the visual sensor coordinate system through the extrinsic parameter matrix of the visual sensor. Then, the data in the visual sensor coordinate system is transformed into the image coordinate system through the intrinsic parameter matrix of the visual sensor. Finally, the target in the image coordinate system is transformed into the pixel coordinate system through the intrinsic parameter matrix of the visual sensor, completing the transformation from target to image.

[0009] Preferably, the conversion function process is as follows: From radar coordinate system to world coordinate system: ; Let be the coordinates of the origin of the radar coordinate system in the world coordinate system. Points in the radar coordinate system The point corresponding to the world coordinate system; From world coordinate system to camera coordinate system: ; This is the angle required to rotate the world coordinate system about the positive x-axis until it is parallel to the camera coordinate system. The coordinates of the origin of the world coordinate system in the camera coordinate system are given. Points in the world coordinate system The corresponding point in the camera coordinate system.

[0010] From camera coordinate system to image coordinate system: ; For camera focal length, Points in the camera coordinate system The corresponding point on the image coordinate system.

[0011] Image coordinate system to pixel coordinate system: ; and For the length and width of the camera's image sensor, and Width and height of the camera's original pixels. and These are the cropped camera pixels.

[0012] Furthermore, the DBSCAN point cloud clustering algorithm is used to convert the detected targets of the millimeter-wave radar into cluster centers. Then, it identifies whether they are the same target by determining whether the cluster center point is within the camera's detection frame. The determination criteria are as follows: ; , The coordinates of the target detection bounding box identified by the camera.

[0013] The system uses information such as distance and angle detected by millimeter-wave radar to compensate for the target position information that is difficult to measure accurately by cameras, and finally outputs the fusion result to provide the lawn mowing robot with decision-making information.

[0014] The beneficial effects of this invention are as follows: The environmental perception method and device for lawn mowing robots based on the fusion of vision and millimeter-wave radar simplify the structure and design complexity of the product to a certain extent. By integrating advanced vision sensors with affordable millimeter-wave radar, the lawn mowing robot achieves intelligent perception of the outdoor working environment, while improving its all-weather operation capability and adaptability to different lighting conditions, as well as enhancing its robustness and positioning accuracy. Furthermore, the introduction of millimeter-wave radar effectively solves the problem of target occlusion, improving the safety of lawn mowing robot operations. Finally, this invention improves the accuracy of target recognition to a certain extent, enabling lawn mowing robots to complete mowing tasks more efficiently. Attached Figure Description

[0015] The above and other features of the present invention will become more apparent from the detailed description of the embodiments shown in conjunction with the accompanying drawings. In the accompanying drawings, the same reference numerals denote the same or similar elements. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without any creative effort. In the drawings: Figure 1 The diagram shows the overall flowchart of the environmental perception method and device for lawn mowing robots based on the fusion of vision and millimeter-wave radar. Figure 2 The image shows the point cloud clustering effect of the DBSCAN algorithm, which is based on the fusion of vision and millimeter-wave radar for the environmental perception method and device of lawn mowing robot. Figure 3 The diagram shows the sensor time synchronization flowchart of the environmental perception method and device for lawn mowing robots based on the fusion of vision and millimeter-wave radar. Figure 4 The diagram shows the sensor spatial synchronization flowchart of an environmental perception method and device for lawn mowing robots based on the fusion of vision and millimeter-wave radar. Detailed Implementation

[0016] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0017] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0018] Furthermore, all information and data (including but not limited to user device information, user personal information, etc.) involved in this application are information and data authorized by the user or fully authorized by all parties. For example, this system has an interface with the relevant user or organization. Before obtaining relevant information, it needs to send an acquisition request to the aforementioned user or organization through the interface, and obtain the relevant information after receiving consent from the aforementioned user or organization.

[0019] Example 1: According to the embodiments of this application, an embodiment of an environmental perception method for a lawn mowing robot based on the fusion of vision and millimeter-wave radar is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Also, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0020] like Figure 1 As shown, step one: Obtain preprocessed visual sensor data. Images of lawns, boundaries, and obstacles are collected using a visual sensor, and some lawn data is collected from the Roboflow website. The images acquired through the visual sensor and the collected lawn data are finely annotated to construct a lawn dataset. The lawn dataset is then trained using the YOLOv11 instance segmentation model. Considering the limited computing resources of embedded chips, the relevant parameters and optimizer of the model are modified before training the lawn dataset using the YOLOv11 instance segmentation model. The modifications include changing the activation function from SiLU to ReLU; changing the model parameter lr0 from 0.01 to 0.001 and lrf from 0.01 to 0.9; and changing the optimizer from SGD to AdamW to prevent overfitting. The parameter lr0 is the initial learning rate, and the parameter lrf is the learning rate factor.

[0021] Step Two: Acquire preprocessed millimeter-wave radar sensor data. Write a radar parsing program according to the millimeter-wave radar communication protocol to obtain the point cloud information of the target. The target is the point cloud information of the lawn obstacle obtained by the millimeter-wave radar. Use the DBSCAN point cloud clustering algorithm to cluster the target's point cloud information, converting the scattered point cloud of the detected target into cluster centers. Use the information of the cluster centers to represent the relevant location information of the detected target. The clustering result is as follows: Figure 2 As shown.

[0022] The above data is synchronized in time and space. The time synchronization steps are as follows: the detection messages of the two sensors are time-aligned based on the timestamp interpolation method. The timestamps of the millimeter-wave radar and the vision sensor are obtained by writing a program and the time difference between them is calculated. Then, the camera data is time-aligned using linear interpolation based on the timestamp of the radar data to ensure that the timestamps of the camera data and the radar data are consistent, thereby obtaining synchronized data. Specifically, such as Figure 3 As shown, to ensure that the timestamps of the camera data and radar data are consistent, thus obtaining synchronized data, the linear interpolation function is as follows: ; , Given data from two known points, Let x be the x-coordinate of the point to be interpolated. for The interpolation function value is obtained at the given location.

[0023] The spatial synchronization steps are as follows: the target position is transformed into the millimeter-wave radar coordinate system, then the millimeter-wave radar coordinate system is transformed into the world coordinate system through the coordinate transformation relationship, then the target data in the world coordinate system is transformed into the visual sensor coordinate system through the extrinsic parameter matrix of the visual sensor, then the data in the visual sensor coordinate system is transformed into the image coordinate system through the intrinsic parameter matrix of the visual sensor, and finally the target in the image coordinate system is transformed into the pixel coordinate system through the intrinsic parameter matrix of the visual sensor, thus completing the transformation from target to image.

[0024] Preferably, the same target is identified by determining whether the cluster center point is within the camera's detection frame, and the fusion result is finally output for the lawn mowing robot to make a decision.

[0025] Specifically, such as Figure 4 As shown, the conversion function flow is as follows: From radar coordinate system to world coordinate system: ; Let be the coordinates of the origin of the radar coordinate system in the world coordinate system. Points in the radar coordinate system The point corresponding to the world coordinate system; From world coordinate system to camera coordinate system: ; This is the angle required to rotate the world coordinate system about the positive x-axis until it is parallel to the camera coordinate system. The coordinates of the origin of the world coordinate system in the camera coordinate system are given. Points in the world coordinate system The corresponding point in the camera coordinate system.

[0026] From camera coordinate system to image coordinate system: ; For camera focal length, Points in the camera coordinate system The corresponding point on the image coordinate system.

[0027] Image coordinate system to pixel coordinate system: ; and For the length and width of the camera's image sensor, and Width and height of the camera's original pixels. and These are the cropped camera pixels.

[0028] Example 2, based on Example 1 of this application, also provides an example of an environmental perception device for a lawn mowing robot based on the fusion of vision and millimeter-wave radar. The device includes: a vision sensor, a millimeter-wave radar sensor, an AI chip, and a machine vision processing system. The visual sensor and the millimeter-wave radar sensor acquire image data and location information respectively, and the acquired image data and location information are processed by the AI ​​chip and the machine vision processing system.

[0029] According to an embodiment of this application, a computer program product is also provided, which includes a stored computer program, wherein when the computer program is executed by a processor, it implements the environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar in Embodiment 1.

[0030] Preferably, the computer program performs the following steps: constructing a lawn dataset, adjusting the relevant parameters of the YOLOv11 instance segmentation model, completing the training and quantization deployment of the YOLOv11 instance segmentation model, and preprocessing the camera data.

[0031] The point cloud data of millimeter-wave radar is analyzed, and the DBSCAN algorithm is used to perform clustering processing of the point cloud, thereby realizing the preprocessing of millimeter-wave radar data.

[0032] Time alignment of two sensors is achieved using software timestamps and linear interpolation, and spatial alignment of the two sensors is completed through spatial position transformation.

[0033] The cluster center points acquired by the millimeter-wave radar are projected onto the visual detection plane of the camera, and the decision information of the two sensors is fused by the relevant positional relationship between the cluster center points and the detection box, and the fusion result is finally output.

[0034] Preferably, the computer program is based on a system, which includes a processor and a memory. When the processor executes the computer program, it can implement the steps of any of the above-described methods for intelligent plant selection and community generation based on a scattering algorithm.

[0035] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete component gate circuits, transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the system, connecting various sub-regions of the system via various interfaces and lines.

[0036] The memory can be used to store the computer programs and / or modules. The processor implements various system functions by running or executing the computer programs and / or modules stored in the memory and by calling data stored in the memory. The memory may mainly include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the mobile phone (such as audio data, phonebook, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0037] Although the invention has been described in considerable detail and particularly with regard to several of the described embodiments, it is not intended to limit itself to any of these details or embodiments or any particular embodiment, thereby effectively covering the intended scope of the invention. Furthermore, the invention has been described above with respect to embodiments foreseeable by the inventors in order to provide a useful description, and non-substantial modifications to the invention that have not yet been foreseen may still represent equivalent modifications.

Claims

1. An environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar, characterized in that, The method includes the following steps: Step 1: Construct a lawn dataset, adjust the relevant parameters of the YOLOv11 instance segmentation model, complete the training and quantization deployment of the YOLOv11 instance segmentation model, and realize the preprocessing of visual sensor data; Step 2: Analyze the millimeter-wave radar point cloud data and use the DBSCAN algorithm to perform clustering processing of the point cloud, thereby realizing the preprocessing of the millimeter-wave radar data; Step 3: Use software timestamps and linear interpolation to achieve time alignment of the two sensors, and complete spatial alignment of the two sensors through spatial position transformation relationship; Step 4: Project the cluster center points acquired by the millimeter-wave radar onto the camera's visual detection plane, and fuse the decision information of the two sensors based on the relevant positional relationship between the cluster center points and the detection boxes, and finally output the fusion result.

2. The environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar according to claim 1, characterized in that, In step one, images of lawns, boundaries, and obstacles are acquired using a visual sensor, and some lawn data is collected from the Roboflow website. The images acquired by the visual sensor and the collected lawn data are then finely annotated to construct a lawn dataset. The lawn dataset is then trained using the YOLOv11 instance segmentation model.

3. The environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar according to claim 1, characterized in that, In step two, a radar parsing program is written according to the millimeter-wave radar communication protocol to obtain the point cloud information of the target. The target is the point cloud information of obstacles in the lawn obtained by the millimeter-wave radar. The DBSCAN point cloud clustering algorithm is used to complete the clustering processing of the point cloud information of the target, converting the scattered point cloud of the detected target into cluster center points, and using the information of the cluster center points to represent the relevant location information of the detected target.

4. The environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar according to claim 1, characterized in that, In step three, software timestamps are used to synchronize visual sensor data and millimeter-wave radar sensor data. Detection messages from both sensors are obtained via a RabbitMQ message queue. Then, the detection messages from the two sensors are time-aligned using a timestamp interpolation method. A program is written to obtain the timestamps of the millimeter-wave radar and visual sensor, calculate the time difference between them, and then use the radar data timestamp as a reference to time-align the camera data using linear interpolation, ensuring that the timestamps of the camera data and radar data are consistent, thus obtaining the synchronized data. The linear interpolation function is as follows: ； , Given data from two known points, Let x be the x-coordinate of the point to be interpolated. for The interpolation function value is obtained at the given location.

5. The environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar according to claim 1, characterized in that, In step four, the location of the obstacle target in the lawn is first detected by millimeter-wave radar. The target position is then converted to the millimeter-wave radar coordinate system. Next, the millimeter-wave radar coordinate system is converted to the world coordinate system through a coordinate transformation relationship. Then, the target data in the world coordinate system is converted to the visual sensor coordinate system through the extrinsic parameter matrix of the visual sensor. Finally, the data in the visual sensor coordinate system is converted to the image coordinate system through the intrinsic parameter matrix of the visual sensor. This completes the conversion from target to image.

6. The environmental perception method for lawn mowing robots based on the fusion of vision and millimeter-wave radar according to claim 1, characterized in that, The DBSCAN point cloud clustering algorithm is used to convert detected targets into cluster centers for millimeter-wave radar. Then, it identifies whether they are the same target by determining whether the cluster center point is within the camera's detection frame. The determination criteria are as follows: ； , The coordinates of the target detection bounding box identified by the camera; The system uses information such as distance and angle detected by millimeter-wave radar to compensate for the target position information that is difficult to measure accurately by cameras, and finally outputs the fusion result to provide the lawn mowing robot with decision-making information.

Images