Obstacle detection method based on radar and vision fusion and automatic mower

Abstract

This invention relates to an obstacle detection method and an automatic lawnmower based on radar and vision fusion. The obstacle detection method first initializes by synchronizing the time and aligning the space of radar and vision methods. Then, it acquires obstacle position information using radar and vision methods respectively. Finally, it fuses the two obstacle position information to obtain the final obstacle information, thereby effectively avoiding the problem that existing lawnmowers cannot avoid obstacles well when vision acquisition is poor.

Description

Technical Field

[0001] This invention relates to the technical field of path planning for self-moving robots, and more particularly to an obstacle detection method based on millimeter radar and vision fusion, and an automatic lawnmower incorporating this method. Background Technology

[0002] Currently, the most common intelligent maintenance equipment used for lawn maintenance and beautification is the automatic lawnmower. Most existing automatic lawnmower robots use sensor and intelligent control technology, integrating functions such as perception, recognition, decision-making and control. They can autonomously mow lawns according to a planned path and return to the charging station to recharge.

[0003] Currently, most automatic lawnmowers rely on visual signals acquired by cameras to detect and avoid obstacles during operation. However, in low light conditions at night, when strong light shines directly on the lens, in reflective environments, or in water areas, the camera may experience delays in image acquisition, untimely dynamic response, insufficient image resolution, or imaging errors, or fail to identify water surfaces. This can lead to incorrect obstacle recognition or obstacle location information, resulting in reduced obstacle avoidance sensitivity and success rate. Summary of the Invention

[0004] This invention provides an obstacle detection method based on radar and vision fusion, which aims to solve the problem that existing lawnmowers cannot effectively avoid obstacles when visual acquisition is poor.

[0005] In a first aspect, embodiments of the present invention provide an obstacle detection method based on radar and vision fusion, comprising: performing time synchronization and joint calibration of a camera and a millimeter-wave radar to obtain the internal parameters of the camera and the external parameters between the camera and the millimeter-wave radar; based on the joint calibration, transforming the point cloud data collected by the millimeter-wave radar from the radar coordinate system to the camera coordinate system to achieve spatial alignment between radar data and visual data; performing model inference based on the image data collected by the camera to identify static obstacles in the image and determine their positions; identifying obstacles and calculating their positions based on the point cloud data generated by the millimeter-wave radar scan; and fusing the obstacles identified by the camera and the obstacles identified by the millimeter-wave radar to obtain and output the fused obstacle position information.

[0006] In a preferred embodiment, the "time synchronization and joint calibration of the camera and millimeter-wave radar" includes: using the clock of the millimeter-wave radar to trigger the camera to acquire image data to achieve time synchronization; calibrating the internal parameters of the camera, including focal length, principal point, and distortion coefficient; and calibrating the external parameters between the camera and the millimeter-wave radar, including rotation matrix and translation matrix.

[0007] In a preferred embodiment, the step of "performing model inference based on image data acquired by the camera, identifying static obstacles in the image and determining their positions" includes: preprocessing the acquired image, the preprocessing including resizing, normalization, and color space conversion; inputting the preprocessed image into a trained detection and segmentation model for inference to obtain an inference result containing obstacle bounding boxes, category labels, and segmentation masks; postprocessing the inference result, the postprocessing including removing duplicate detection boxes through non-maximum suppression and performing morphological operations on the segmentation result; and determining the positions of static obstacles in the image based on the postprocessing result.

[0008] In a preferred embodiment, the step of "identifying obstacles and calculating their positions based on the point cloud data generated by the millimeter-wave radar scan" includes: filtering the original point cloud data of the millimeter-wave radar to remove clutter and noise; extracting effective obstacles from the filtered point cloud data; and calculating the distance, speed, and angle information of the effective obstacles.

[0009] In a preferred embodiment, the step of "data fusion of obstacles identified by the camera and obstacles identified by the millimeter-wave radar" includes: projecting the obstacle point cloud identified by the millimeter-wave radar onto the plane where the camera image is located; matching and associating the projected radar point cloud with the obstacle bounding boxes and category labels identified by the camera; and fusing the successfully matched radar obstacle information with the visual obstacle information to generate a final obstacle list containing location, category, speed, and angle.

[0010] In a preferred embodiment, after the successfully fused radar obstacle information and visual obstacle information are fused, the obstacle detection method further includes: using a Kalman filter to perform multi-frame data fusion and trajectory smoothing on the obstacles in the final obstacle list, so as to improve the stability and prediction accuracy of obstacle tracking.

[0011] In a preferred embodiment, the step of "converting the point cloud data acquired by the millimeter-wave radar from the radar coordinate system to the camera coordinate system" includes: using the rotation matrix and translation matrix in the external parameters, the point cloud data is transformed from the radar coordinate system to the lawnmower coordinate system through coordinate transformation, and then transformed from the lawnmower coordinate system to the camera coordinate system.

[0012] In a preferred embodiment, "using the rotation and translation matrices in the external parameters, transforming the point cloud data from the radar coordinate system to the lawnmower coordinate system, and then from the lawnmower coordinate system to the camera coordinate system" includes: The three-dimensional coordinates of the point cloud data The coordinates are converted into lawnmower coordinates using a first preset matrix, which is as follows: = In the formula, R represents the coordinate value in the lawnmower coordinate system. v2r T represents the rotation matrix between the lawnmower coordinate system and the millimeter-wave radar coordinate system. v2r This represents the translation matrix between the lawnmower coordinate system and the millimeter-wave radar coordinate system. The coordinate values ​​in the lawnmower coordinate system are converted into coordinate values ​​in the camera coordinate system using a second preset matrix, wherein the second preset matrix is ​​as follows: = In the formula, R represents the coordinate value in the camera coordinate system. v2c T represents the rotation matrix between the lawnmower coordinate system and the camera coordinate system. v2c This represents the translation matrix between the lawnmower coordinate system and the camera coordinate system. The coordinate values ​​in the camera coordinate system are converted into coordinate values ​​in the pixel coordinate system using a third preset matrix, wherein the third preset matrix is ​​as follows: = In the formula, R represents the homogeneous coordinate values ​​in the pixel coordinate system. in This represents the intrinsic parameter matrix of the camera; The coordinate values ​​in the pixel coordinate system are normalized to obtain the pixel coordinate values ​​projected from the point cloud data onto the plane of the camera image. The normalization process is as follows: = In the formula, This represents the width value of the projection of the point cloud data. This represents the height value of the projection of the point cloud data.

[0013] In another aspect, embodiments of the present invention also provide an automatic lawnmower, including a body, wheels disposed under the body, a camera disposed on the body for acquiring environmental images, a radar disposed on the body, and a controller disposed inside the body. The controller is communicatively connected to the camera, the radar, and the wheels, respectively. The controller is characterized in that it detects obstacles on the movement path of the automatic lawnmower according to the obstacle detection method described above, and adjusts the movement path to avoid the obstacles based on the obstacle position information output by the obstacle detection method.

[0014] This invention improves the accuracy of obstacle identification and increases the obstacle avoidance success rate of automatic lawnmowers by combining visual obstacle information with radar information to supplement and confirm the location information. This is achieved by using radar to supplement the location information based on visual information, which can compensate for situations where vision cannot identify obstacles or may identify them incorrectly in some special circumstances. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating an obstacle detection method based on radar and vision fusion, provided as an embodiment of the present invention.

[0016] Figure 2 Applications provided in the embodiments of the present invention Figure 1 A schematic diagram of the working scenario of the automatic lawnmower using the method shown. Detailed Implementation

[0017] To further illustrate the technical means and effects adopted by the present invention to achieve the intended inventive objective, the specific embodiments, structures, and features according to the present invention will be described below in conjunction with the accompanying drawings and preferred embodiments. Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, exemplary embodiments can be implemented in various forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this disclosure more comprehensive and complete, and to fully convey the concept of exemplary embodiments to those skilled in the art.

[0018] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this disclosure. However, those skilled in the art will recognize that the technical solutions of this disclosure can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this disclosure.

[0019] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software, in one or more hardware modules or integrated circuits, or in different network and / or processor devices and / or microcontroller devices.

[0020] The flowchart shown in the attached diagram is merely illustrative and does not necessarily include all content and operations / steps, nor does it require execution in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances. Its effects are explained in detail below.

[0021] Millimeter-wave radar point cloud data is a dataset of obstacle point clouds obtained by millimeter-wave radar equipment emitting millimeter waves for spatial detection. Each point contains information such as the target's distance, velocity, angle, and radar cross-section (RCS). In practice, millimeter-wave radar point cloud data is generated by the millimeter-wave radar equipment emitting electromagnetic wave signals with wavelengths between 1-10 mm, then receiving the electromagnetic wave signals reflected by obstacles, and processing and calculating the data to obtain information such as the spatial motion of the point cloud. Furthermore, cameras can be used to achieve various functions. Cameras equipped on automatic lawnmowers can acquire sufficient environmental details to help the automatic lawnmower recognize its surroundings; cameras can also depict the appearance and shape of objects, read signs, etc. Under good weather and lighting conditions, cameras are excellent sensors for external environmental perception.

[0022] like Figure 1 As shown in the figure, an obstacle detection method based on radar and vision fusion is provided by an embodiment of the present invention. The detection method specifically includes the following steps.

[0023] Step S1: Perform time synchronization and joint calibration of the camera and millimeter-wave radar to obtain the internal parameters of the camera and the external parameters between the camera and the millimeter-wave radar.

[0024] Precise time synchronization of the camera and millimeter-wave radar, two different sensor devices, ensures their consistency across time, enabling accurate data fusion later. Simultaneously, joint calibration of the camera and millimeter-wave radar allows for the acquisition of the camera's internal parameters, including but not limited to focal length, principal point coordinates, and lens distortion coefficients. Furthermore, acquiring the external parameters between the camera and millimeter-wave radar is crucial. These external parameters describe their relative spatial position and angular attitude, essential for achieving collaborative operation and data fusion.

[0025] Step S2: Based on the joint calibration, the point cloud data collected by the millimeter-wave radar is transformed from the radar coordinate system to the camera coordinate system to achieve spatial alignment between radar data and visual data.

[0026] Based on the joint calibration, the point cloud data acquired by the millimeter-wave radar was transformed from its original radar coordinate system to a camera coordinate system that matches the camera data through a series of precise mathematical transformations. By spatially aligning the data acquired by the two different sensors, the depth and distance information acquired by the millimeter-wave radar can be fused and matched with the visual image information captured by the camera within the same spatial reference frame. This achieves precise spatial alignment between radar and visual data, laying a solid foundation for further analysis and processing of multi-sensor data.

[0027] Step S3: Based on the image data collected by the camera, perform model reasoning to identify static obstacles in the image and determine their positions.

[0028] In this step, model reasoning is carried out using only the image data collected by the camera to identify static obstacles in the image and determine the specific location of the static obstacles in the actual environment.

[0029] Step S4: Based on the point cloud data generated by the millimeter-wave radar scan, identify the obstacles and calculate their positions.

[0030] In this step, millimeter-wave radar is used solely as the sensor. Based on the point cloud data generated from its high-precision scanning of the surrounding environment, this data is further analyzed and processed to accurately identify obstacles and their locations. Millimeter-wave radar, due to its strong penetration capability and anti-interference characteristics, can operate stably in various complex environments. The point cloud data it generates contains a wealth of spatial information and reflection intensity characteristics. By applying algorithms such as filtering, clustering, and classification to this data, key information such as the shape, position, and motion state of obstacles can be effectively extracted, providing a reliable basis for subsequent decision-making and control.

[0031] Step S5: Perform data fusion between the obstacles identified by the camera and the obstacles identified by the millimeter-wave radar to obtain and output the fused obstacle location information.

[0032] This invention improves the accuracy of obstacle identification and increases the obstacle avoidance success rate of automatic lawnmowers by combining visual obstacle information with radar information to supplement and confirm the location information. This is achieved by using radar to supplement the location information based on visual information, which can compensate for situations where vision cannot identify obstacles or may identify them incorrectly in some special circumstances.

[0033] Specifically, step S1, "time synchronization and joint calibration of the camera and millimeter-wave radar", includes the following steps S11 to S13.

[0034] Step S11: Use the millimeter-wave radar's clock to trigger the camera to acquire image data, achieving time synchronization. By using the millimeter-wave radar's clock signal as the trigger source for camera image acquisition, it ensures that the camera acquires image data of the corresponding scene at the same timestamp when the radar emits electromagnetic waves and receives the echo signal. This eliminates the time difference caused by the drift of the two independent operating clocks, ensuring precise alignment of radar point cloud data and camera image data in the time dimension, and providing a time reference consistency guarantee for subsequent data fusion.

[0035] Step S12: Calibrate the internal parameters of the camera, including focal length, principal point, and distortion coefficient.

[0036] Specifically, this is achieved using Zhang's calibration method. First, a calibration board containing several black and white checkerboard patterns is prepared. This board is then photographed multiple times within the camera's field of view at different poses and positions, acquiring at least 10 images containing complete checkerboard patterns. Corner detection is performed on each image, extracting the pixel coordinates of the checkerboard corner points in the image coordinate system. The 3D coordinates of the corner points in the world coordinate system are then determined based on the actual physical dimensions of the calibration board. Next, a correspondence between the image coordinates and world coordinates is established based on a perspective projection model. The camera's intrinsic parameter matrix (including focal lengths f_x, f_y and principal point coordinates u_0, v_0) and distortion coefficients (including radial distortion coefficients k1, k2, k3 and tangential distortion coefficients p1, p2) are solved using the least squares method. After calibration, the obtained intrinsic parameters are used to correct the distortion of the original images acquired by the camera, eliminating image distortion caused by lens optical characteristics and ensuring the accuracy of subsequent image feature extraction and coordinate transformation.

[0037] Step S13: Calibrate the external parameters between the camera and the millimeter-wave radar, the external parameters including the rotation matrix and the translation matrix.

[0038] Specifically, this can be achieved using either hand-eye calibration or a joint calibration method based on a calibration board. Taking the joint calibration method based on a calibration board as an example, the calibration board is first fixed within the common field of view of both the radar and the camera, ensuring that the calibration board appears simultaneously in the image captured by the camera and the detection range of the millimeter-wave radar. For the camera, its intrinsic parameters have already been obtained using the Zhang's calibration method described above, so the corner pixel coordinates of the calibration board can be directly detected and extracted from the image. For the millimeter-wave radar, the plane where the calibration board is located is identified by clustering and plane fitting of its original point cloud data, and the three-dimensional coordinates of the calibration board corners in the radar coordinate system are extracted. Then, based on the coordinate correspondence between the same calibration board corners in the camera image coordinate system and the radar coordinate system, a coordinate transformation equation is constructed, and algorithms such as singular value decomposition (SVD) are used to solve for the rotation matrix R and translation matrix T from the camera coordinate system to the radar coordinate system. During the solution process, joint optimization of multiple sets of calibration board data at different positions and attitudes is required to reduce the influence of random errors. Once the external parameters are obtained, the image data captured by the camera and the point cloud data of the millimeter-wave radar can be spatially aligned in the same coordinate system, laying the foundation for subsequent sensor data fusion.

[0039] In step S2, the spatial fusion of the millimeter-wave radar and the camera aims to correlate objects in the three-dimensional world detected by the millimeter-wave radar with objects in the images detected by the camera. Since the millimeter-wave radar and the camera are sensors in different coordinate systems, a transformation model between the coordinate systems of the two sensors must be established to achieve spatial fusion. This embodiment specifically involves four coordinate systems: the millimeter-wave radar coordinate system, the lawnmower coordinate system, the camera coordinate system, and the coordinate value conversion to the pixel coordinate system. The step of "converting the point cloud data collected by the millimeter-wave radar from the radar coordinate system to the camera coordinate system" specifically involves: using the rotation and translation matrices in the external parameters, the point cloud data is transformed from the radar coordinate system to the lawnmower coordinate system, and then from the lawnmower coordinate system to the camera coordinate system through coordinate transformation. Specifically, the coordinate system transformation process is as follows: Step S21: Convert the three-dimensional coordinates of the point cloud data The coordinates are converted into lawnmower coordinates using a first preset matrix, which is as follows: = In the formula, R represents the coordinate value in the lawnmower coordinate system. v2r T represents the rotation matrix between the lawnmower coordinate system and the millimeter-wave radar coordinate system. v2r This represents the translation matrix between the lawnmower coordinate system and the millimeter-wave radar coordinate system. Specifically, the three-dimensional coordinates detected by the millimeter-wave radar are first transformed to the lawnmower coordinate system. Due to the rotation matrix R...v2r It is used to rotate from the lawnmower coordinate system to the millimeter-wave radar coordinate system. Therefore, when using a rotation matrix to switch from the millimeter-wave radar coordinate system to the lawnmower coordinate system, the inverse of the rotation matrix must be used.

[0040] Step S22: Convert the coordinate values ​​in the lawnmower coordinate system into coordinate values ​​in the camera coordinate system using a second preset matrix, wherein the second preset matrix is ​​as follows: = In the formula, R represents the coordinate value in the camera coordinate system. v2c T represents the rotation matrix between the lawnmower coordinate system and the camera coordinate system. v2c This represents the translation matrix between the lawnmower coordinate system and the camera coordinate system. Specifically, when transforming from the lawnmower coordinate system to the camera coordinate system, the translation matrix must be calculated first; otherwise, the values ​​in the translation matrix will also be affected by the rotation matrix, thus affecting the projection result.

[0041] Step S23: Convert the coordinate values ​​in the camera coordinate system into coordinate values ​​in the pixel coordinate system using a third preset matrix, wherein the third preset matrix is ​​as follows: = In the formula, R represents the homogeneous coordinate values ​​in the pixel coordinate system. in This represents the intrinsic parameter matrix of the camera. Specifically, the transformation from millimeter-wave radar data to camera coordinates, and from the camera coordinate system to pixel coordinates, requires the camera's intrinsic parameter matrix Rin. The most commonly used method for calculating the camera's intrinsic parameter matrix is ​​the Zhang Zhengyou calibration method. The Zhang Zhengyou calibration method is a linear calibration method for nonlinear model cameras. It uses a two-dimensional planar target to acquire images from multiple different viewpoints, thus achieving camera calibration. The Zhang Zhengyou calibration method is chosen to calibrate the camera and obtain relevant parameters. It is characterized by its simplicity, adaptability, and high calibration accuracy.

[0042] Step S23: Normalize the coordinate values ​​in the pixel coordinate system to obtain the pixel coordinate values ​​projected from the point cloud data onto the plane of the camera image. The normalization process is as follows: = In the formula, This represents the width value of the projection of the point cloud data. This represents the height value of the projection of the point cloud data. (During normalization processing...) The pixel coordinates are a homogeneous representation, but they need to be normalized to obtain a two-dimensional plane, which represents the pixel coordinate values ​​projected by the millimeter-wave radar. This embodiment uses normalization to further improve obstacle detection accuracy.

[0043] Specifically, step S3, "performing model reasoning based on the image data collected by the camera, identifying static obstacles in the image and determining their positions," includes the following steps S31 to S33.

[0044] Step S31: Preprocess the acquired image, including size adjustment, normalization and color space conversion.

[0045] The preprocessing steps include: resizing to scale the image to the required resolution for the model input, such as 224×224 pixels, using bilinear interpolation to preserve image details while reducing computation; normalization to convert pixel values ​​from the integer range of [0,255] to the floating-point range of [-1,1] or [0,1] to match the input distribution of the deep learning model and avoid gradient explosion caused by excessively large pixel values; and color space conversion to convert the original RGB color space to HSV or YCrCb color space to enhance robustness to changes in illumination. For example, in HSV space, the hue (H), saturation (S), and lightness (V) channels can be separated, and the contrast of the lightness channel can be adjusted separately to reduce the impact of uneven illumination on subsequent feature extraction. After preprocessing, the image data will meet the input format requirements of the model inference, providing a high-quality image foundation for subsequent static obstacle recognition.

[0046] Step S32: Input the preprocessed image into the trained detection and segmentation model for inference to obtain the inference result containing obstacle bounding boxes, category labels and segmentation masks.

[0047] Specifically, the detection and segmentation model employs an improved YOLOv5 architecture, introducing a CSPDarknet53 structure into the backbone network to enhance feature extraction capabilities. A multi-scale feature fusion module (PANet) is used to accurately capture obstacles of different sizes. During inference, the model first constructs a feature pyramid from the preprocessed image, extracting detailed information such as edges and textures from the low-level feature map. After upsampling and fusion with high-level semantic features, the detection head outputs the bounding box coordinates (x, y, w, h), confidence scores, and class probability distributions of the obstacles. For segmentation mask generation, the model introduces a mask branch in parallel with the detection branch, using dynamic convolutional kernels to perform pixel-level classification of the region of interest, ultimately outputting a binarized mask image that perfectly matches the obstacle contours. To improve inference efficiency, the model also employs TensorRT acceleration technology, compressing the inference time per image to less than 20ms through layer fusion, accuracy calibration, and other optimization methods, meeting the real-time requirements of automatic lawnmowers.

[0048] Step S33: Post-process the inference result, which includes removing duplicate detection boxes by non-maximum suppression and performing morphological operations on the segmentation result; determine the position of static obstacles in the image based on the post-processing result.

[0049] Specifically, in the non-maximum suppression process, the detection boxes are first sorted according to their confidence scores, and the box with the highest score is selected as the baseline box. The intersection-over-union (IoU) ratio between the baseline box and the other candidate boxes is calculated. When the IoU value is greater than a set threshold (e.g., 0.5), the two boxes are considered to belong to the same obstacle, and the boxes with lower confidence scores are removed. This process is repeated iteratively to effectively eliminate redundant boxes generated by the model detecting the same target multiple times, ensuring that only one optimal detection box is retained for each static obstacle. For morphological operations on the segmentation results, erosion and dilation are mainly used: first, a 3x3 structuring element is used to erode the binary mask image to remove small noise points and isolated pixels at the segmentation boundary; then, dilation is performed to restore the complete outline of the main obstacle region, avoiding target shape distortion caused by excessive erosion, and making the segmentation mask more accurately reflect the actual shape of the static obstacle. After post-processing, by combining the bounding box coordinates (x, y, w, h) of the detection boxes with the pixel-level position information of the segmentation mask, the two-dimensional planar position of the static obstacle in the image can be accurately located, laying the foundation for subsequent fusion with radar data.

[0050] Furthermore, the step S4, "identifying obstacles and calculating their positions based on the point cloud data generated by the millimeter-wave radar scan," includes the following steps S41 to S42.

[0051] Step S41: Filter the raw point cloud data from the millimeter-wave radar to remove clutter and noise. In a specific embodiment, the filtering process can employ an adaptive radius filtering algorithm. First, the search radius is dynamically adjusted based on the density distribution of the point cloud data. For each point, the number of its neighboring points within a set radius is counted. When the number of neighboring points is less than a preset threshold, the point is determined to be an isolated noise point and is removed. Simultaneously, height threshold filtering is combined with this method. By setting a reasonable height range, ground reflection points and suspended interference points in the air are filtered out, retaining the effective obstacle point cloud within the lawnmower's working height range. After filtering, noise interference in the point cloud data is significantly reduced, providing a more reliable data foundation for subsequent obstacle recognition.

[0052] Step S42: Extract effective obstacles from the filtered point cloud data and calculate the distance, speed and angle information of the effective obstacles.

[0053] In practice, the filtered point cloud data is first subjected to cluster analysis using the density-based DBSCAN (Density-Based Spatial Clustering of Applications with Noise) algorithm. By setting two core parameters—neighborhood radius and minimum number of contained points—points that are spatially close and meet the density requirements are aggregated into a single obstacle candidate cluster, thus distinguishing different obstacle targets. For each clustered obstacle candidate cluster, the coordinates of its center point are calculated and used as a reference point for the obstacle's position. Then, based on the Doppler effect principle of millimeter-wave radar, the radial velocity information of each point is extracted from the point cloud data, and the velocities of all points belonging to the same obstacle candidate cluster are statistically averaged to obtain the overall moving speed of the obstacle. Simultaneously, based on the scanning angle information of the millimeter-wave radar and the coordinate position of the obstacle's center point in the radar coordinate system, the azimuth angle of the obstacle relative to the lawnmower is calculated using trigonometric relationships. Through the above processing, effective obstacles in point cloud data can be accurately identified, and their straight-line distance from the lawnmower, relative speed, and horizontal azimuth angle can be precisely obtained, providing key obstacle state parameters for subsequent obstacle avoidance decisions.

[0054] In one specific embodiment, the step S5 of "fusing data between the obstacle identified by the camera and the obstacle identified by the millimeter-wave radar" includes steps S51 to S53 or steps S51 to S54.

[0055] Step S51: Project the obstacle point cloud identified by the millimeter-wave radar onto the plane where the camera image is located.

[0056] Specifically, the first step is to establish the transformation relationship between the millimeter-wave radar coordinate system and the camera image coordinate system. Given the installation position parameters of the millimeter-wave radar (including translation and rotation angles relative to the camera) and the camera's intrinsic parameters (such as focal length and principal point coordinates), the three-dimensional coordinates of the obstacle point cloud in the radar coordinate system are transformed using coordinate transformation formulas. Convert to 3D coordinates in camera coordinate system Subsequently, using the perspective projection principle of the camera, the three-dimensional coordinates in the camera coordinate system were... Projecting onto the image plane yields the corresponding image pixel coordinates. The projection process must consider the effects of factors such as lens distortion and perform corresponding distortion corrections to ensure the accuracy of the projected point's position on the image. Through this series of coordinate transformations and projection calculations, a precise mapping of the millimeter-wave radar obstacle point cloud to the camera image plane is achieved, providing a spatial basis for subsequent obstacle matching.

[0057] Step S52: Match and associate the projected radar point cloud with the obstacle bounding boxes and category labels identified by the camera.

[0058] When performing matching and association, the first step is to use the projected pixel coordinates of the radar point cloud on the image plane obtained in step S51. , The system determines whether the projected point falls within a bounding box of an obstacle identified by the camera's target detection algorithm. If the projected point is within a bounding box, a preliminary spatial correspondence between the radar point and the obstacle in the image is established. To improve matching accuracy, obstacle category information is used for auxiliary judgment. For example, if the radar point cloud indicates the presence of an obstacle with a certain height and width, and the corresponding projected area in the image identifies the obstacle as a "rock" or "tree trunk," the reliability of the match is high. Conversely, if the corresponding area in the image identifies an aerial target such as a "bird," while the radar point cloud shows it is close to the ground and has a large radial velocity, a mismatch may exist, requiring further verification. Furthermore, for cases where the same obstacle may correspond to multiple radar point clouds, the projected points of these point clouds need to be clustered. If multiple projected points fall within the same image obstacle bounding box, the reliability of the association is enhanced. By using the dual verification of position constraints and category information, the radar point cloud and the image obstacle bounding box and category label are accurately matched and associated, thereby effectively fusing the precise information such as distance and speed detected by the radar with the rich category and texture information provided by the camera.

[0059] Step S53: Merge the successfully matched radar obstacle information with visual obstacle information to generate a final obstacle list containing location, category, speed, and angle.

[0060] Specifically, the system first extracts precise three-dimensional position coordinates (such as x, y, and z axis coordinates), relative velocity vectors, and azimuth and elevation angles detected by the radar from the successfully matched radar obstacle information. Simultaneously, it extracts obstacle category labels (such as "pedestrian," "pet," "tree trunk," and "rock"), two-dimensional bounding box dimensions, and image semantic segmentation features from the corresponding visual obstacle information. Then, a pre-defined data fusion algorithm is used to integrate the two types of information. For example, the three-dimensional position data provided by the radar serves as the benchmark for obstacle spatial positioning, the visual category labels serve as the core basis for obstacle attribute judgment, velocity information is used for dynamic obstacle motion trend analysis, and angle information helps determine the relative orientation of the obstacle and the automatic lawnmower. For cases with data redundancy or minor conflicts (such as slight deviations between radar and vision in the lateral distance of obstacles), weighted averaging or Kalman filtering methods are used for data calibration to ensure that each item in the final obstacle list contains complete and accurate multi-dimensional information, providing comprehensive data support for the automatic lawnmower's path planning, obstacle avoidance decisions, and motion control.

[0061] Preferably, after the successfully fused radar obstacle information and visual obstacle information are fused, the obstacle detection method further includes step S54: using a Kalman filter to perform multi-frame data fusion and trajectory smoothing on the obstacles in the final obstacle list, so as to improve the stability and prediction accuracy of obstacle tracking.

[0062] In practice, the Kalman filter first constructs system state equations and observation equations based on the obstacle states (such as position and velocity) obtained from the fusion of the current frame. The state equations describe the obstacle's motion over continuous time (such as uniform motion or uniform acceleration), while the observation equations relate the predicted state to the actual fused observations. Upon input of each new frame, the filter first predicts the obstacle's state at the next moment using the state equations and calculates the prediction covariance matrix to characterize the prediction uncertainty. Subsequently, it compares the fused observation data of the current frame with the predicted state, uses the Kalman gain to correct the predicted value, and obtains the updated optimal state estimate. Simultaneously, it updates the covariance matrix to reflect the corrected uncertainty. For obstacles that are temporarily occluded or for which observation data is missing, the Kalman filter can maintain tracking through historical motion trajectories and state prediction models, preventing sudden disappearance of obstacles or trajectory jumps due to single-frame data loss. Through iterative optimization of multi-frame data, this processing step can effectively smooth noise interference in the obstacle's trajectory and reduce the impact of instantaneous observation errors on the tracking results. Especially in complex lawn environments, when obstacles (such as low shrubs or moving pets) exhibit non-uniform motion or partial occlusion, it can significantly improve the estimation accuracy of parameters such as obstacle position and speed, providing more reliable temporal motion information support for the subsequent dynamic obstacle avoidance path planning of the automatic lawnmower (such as early deceleration and detour angle calculation).

[0063] like Figure 2 As shown, this embodiment of the invention also provides an automatic lawnmower 100. The automatic lawnmower 100 includes a body 10, wheels 20 disposed below the body 10, a camera 30 disposed on the body 10 for acquiring environmental images, a radar 40 disposed on the body 10, and a controller (not shown) disposed within the body 10. The controller is communicatively connected to the camera 30, radar 40, and wheels 20. Further, the controller detects obstacles 200 on the movement path of the automatic lawnmower 100 according to the obstacle detection method described above, and adjusts the movement path to avoid the obstacles 200 based on the position information of the obstacles 200 output by the obstacle detection method described above.

[0064] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention in any way. Although the present invention has been disclosed above with reference to preferred embodiments, it is not intended to limit the present invention. Any person skilled in the art can make some modifications or alterations to the above-disclosed technical content to create equivalent embodiments without departing from the scope of the present invention. Any simple modifications, equivalent changes and alterations made to the above embodiments based on the technical essence of the present invention without departing from the scope of the present invention shall still fall within the scope of the present invention.

Claims

1. An obstacle detection method based on radar and vision fusion, characterized in that, The method includes: The system performs time synchronization and joint calibration between the camera and the millimeter-wave radar to obtain the camera's internal parameters and the external parameters between the camera and the millimeter-wave radar. Based on the joint calibration, the point cloud data collected by the millimeter-wave radar is transformed from the radar coordinate system to the camera coordinate system to achieve spatial alignment between radar data and visual data. Based on the image data collected by the camera, model inference is performed to identify static obstacles in the image and determine their positions. Based on the point cloud data generated by the millimeter-wave radar scan, obstacles are identified and their positions are calculated. The obstacles identified by the camera and the obstacles identified by the millimeter-wave radar are fused to obtain and output the fused obstacle position information.

2. The obstacle detection method based on radar and vision fusion according to claim 1, characterized in that, The "time synchronization and joint calibration of the camera and millimeter-wave radar" includes: using the clock of the millimeter-wave radar to trigger the camera to acquire image data to achieve time synchronization; calibrating the internal parameters of the camera, including focal length, principal point and distortion coefficient; calibrating the external parameters between the camera and the millimeter-wave radar, including rotation matrix and translation matrix.

3. The obstacle detection method based on radar and vision fusion according to claim 1 or 2, characterized in that, The step of "performing model inference based on image data acquired by the camera to identify static obstacles in the image and determine their positions" includes: preprocessing the acquired image, including resizing, normalization, and color space conversion; inputting the preprocessed image into a trained detection and segmentation model for inference to obtain inference results containing obstacle bounding boxes, category labels, and segmentation masks; postprocessing the inference results, including removing duplicate detection boxes through non-maximum suppression and performing morphological operations on the segmentation results; and determining the positions of static obstacles in the image based on the postprocessing results.

4. The obstacle detection method based on radar and vision fusion according to claim 1, characterized in that, The step of "identifying obstacles and calculating their positions based on the point cloud data generated by the millimeter-wave radar scan" includes: filtering the original point cloud data of the millimeter-wave radar to remove clutter and noise; extracting effective obstacles from the filtered point cloud data; and calculating the distance, speed, and angle information of the effective obstacles.

5. The obstacle detection method based on radar and vision fusion according to claim 1, characterized in that, The step of "fusing data between obstacles identified by the camera and obstacles identified by the millimeter-wave radar" includes: projecting the obstacle point cloud identified by the millimeter-wave radar onto the plane where the camera image is located; matching and associating the projected radar point cloud with the obstacle bounding boxes and category labels identified by the camera; and fusing the successfully matched radar obstacle information with the visual obstacle information to generate a final obstacle list containing location, category, speed, and angle.

6. The obstacle detection method based on radar and vision fusion according to claim 5, characterized in that, After the successfully fused radar obstacle information and visual obstacle information are fused, the obstacle detection method further includes: using a Kalman filter to perform multi-frame data fusion and trajectory smoothing on the obstacles in the final obstacle list, so as to improve the stability and prediction accuracy of obstacle tracking.

7. The obstacle detection method based on radar and vision fusion according to claim 1, characterized in that, The step of "converting the point cloud data acquired by the millimeter-wave radar from the radar coordinate system to the camera coordinate system" includes: using the rotation matrix and translation matrix in the external parameters, the point cloud data is transformed from the radar coordinate system to the lawnmower coordinate system through coordinate transformation, and then transformed from the lawnmower coordinate system to the camera coordinate system.

8. The obstacle detection method based on radar and vision fusion according to claim 7, characterized in that, "Using the rotation and translation matrices in the external parameters, the point cloud data is transformed from the radar coordinate system to the lawnmower coordinate system, and then from the lawnmower coordinate system to the camera coordinate system through coordinate transformation," including: The three-dimensional coordinates of the point cloud data The coordinates are converted into lawnmower coordinates using a first preset matrix, which is as follows: = In the formula, R represents the coordinate value in the lawnmower coordinate system. v2r T represents the rotation matrix between the lawnmower coordinate system and the millimeter-wave radar coordinate system. v2r This represents the translation matrix between the lawnmower coordinate system and the millimeter-wave radar coordinate system. The coordinate values ​​in the lawnmower coordinate system are converted into coordinate values ​​in the camera coordinate system using a second preset matrix, wherein the second preset matrix is ​​as follows: = In the formula, R represents the coordinate value in the camera coordinate system. v2c T represents the rotation matrix between the lawnmower coordinate system and the camera coordinate system. v2c This represents the translation matrix between the lawnmower coordinate system and the camera coordinate system. The coordinate values ​​in the camera coordinate system are converted into coordinate values ​​in the pixel coordinate system using a third preset matrix, wherein the third preset matrix is ​​as follows: = In the formula, R represents the homogeneous coordinate values ​​in the pixel coordinate system. in This represents the intrinsic parameter matrix of the camera; The coordinate values ​​in the pixel coordinate system are normalized to obtain the pixel coordinate values ​​projected from the point cloud data onto the plane of the camera image. The normalization process is as follows: = In the formula, This represents the width value of the projection of the point cloud data. This represents the height value of the projection of the point cloud data.

9. An automatic lawnmower, comprising a body, wheels disposed under the body, a camera disposed on the body for acquiring environmental images, a radar disposed on the body, and a controller disposed within the body, the controller being communicatively connected to the camera, the radar, and the wheels, characterized in that, The controller detects obstacles on the movement path of the automatic lawnmower according to the obstacle detection method according to any one of claims 1 to 8, and adjusts the movement path to avoid the obstacles based on the obstacle position information output by the obstacle detection method.

Images