Automatic calibration method and system for external parameters of multiple sensors of mowing robot

Abstract

The invention discloses a mowing robot multi-sensor external parameter automatic calibration method and system, and the method comprises the steps: 1, enabling a mowing robot to enter a calibration working mode, and enabling the view of an RGB-D camera to cover a calibration board; 2, the mowing robot executes a preset motion sequence, an RGB-D camera synchronously collects RGB images and depth information in the motion process, and an inertial measurement unit collects IMU data; 3, the RGB-D camera detects calibration plate angular points in the RGB image, and 3D coordinates of the angular points in a camera coordinate system are calculated in combination with depth information; step 4, calculating relative motion between adjacent image frames according to the angular velocity and acceleration of the collected IMU data, and taking the relative motion as a motion constraint of parameter optimization; and step 5, on the basis of visual geometric constraints and motion constraints, constructing an external parameter optimization model by taking minimization of a re-projection error and an IMU motion consistency error as targets, and outputting an external parameter matrix to realize automatic calibration of multi-sensor external parameters. According to the invention, the calibration process is greatly simplified, and the calibration accuracy and efficiency are improved.

Description

Technical Field

[0001] This invention relates to the field of lawnmower technology, and in particular to a method and system for automated calibration of multi-sensor extrinsic parameters of lawnmower robots. Background Technology

[0002] With the rapid development of intelligent technologies, intelligent lawnmowers, as an indispensable part of modern gardening management, have gradually received widespread attention. These robots achieve efficient environmental perception and autonomous navigation by combining visual sensors and inertial measurement units (IMUs). Although existing lawnmowers perform well in practical applications, certain challenges remain in the accuracy of motion estimation and environmental understanding, mainly in the following aspects: (1) Use an RGB-D camera to acquire a real 3D model of the environment and effectively perform path planning and obstacle detection to ensure safe and efficient movement. However, the accuracy of the RGB-D camera is affected by a variety of factors, including changes in ambient lighting, the reflective properties of object surfaces, and the noise of the sensor itself, which may lead to positioning errors in dynamic environments.

[0003] (2) IMUs can accurately determine the robot's posture by accumulating acceleration and rotation data, providing support for path planning. However, IMU data drifts over time, generating cumulative errors and leading to inaccurate long-term motion estimation. This phenomenon is common in many applications, thus requiring effective compensation and correction.

[0004] (3) To improve the positioning and navigation accuracy of lawnmower robots in large-scale complex outdoor environments, RGB-D data is usually fused with IMU information. This fusion can effectively compensate for the shortcomings of a single sensor and improve the robustness and accuracy of the overall system. However, the spatial pose relationship between the RGB-D camera and the IMU needs to be precisely calibrated, but traditional calibration methods usually require manual intervention, which is cumbersome, complex, and prone to human error. In addition, most existing methods require calibration at a fixed position, which cannot adapt to the free movement of the robot in the natural environment. Since traditional calibration methods take a long time, they often cannot meet the needs of real-time applications. This limits the autonomous working ability of lawnmower robots in complex environments. Summary of the Invention

[0005] The purpose of this invention is to overcome the shortcomings of existing calibration methods for lawn mowing robots, which involve cumbersome and complex processes, are prone to human error, and cannot adapt to the free movement of robots in natural environments. This invention provides an automated calibration method and system for multi-sensor extrinsic parameters of lawn mowing robots.

[0006] The objective of this invention is achieved through the following technical solution: An automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot includes the following steps: Step 1: The lawnmower enters calibration mode, and the RGB-D camera's field of view covers the calibration board; Step 2: The lawnmower robot executes a preset motion sequence. During the motion, the RGB-D camera synchronously acquires RGB images and depth information, and the inertial measurement unit acquires IMU data. Step 3: The RGB-D camera detects the calibration board corner points in the RGB image, and calculates the 3D coordinates of the corner points in the camera coordinate system by combining the depth information. The reprojection consistency of the corner points observed by the camera in multiple frames of images is used as the visual geometric constraint for parameter optimization. Step 4: Calculate the relative motion between adjacent image frames using the angular velocity and acceleration of the acquired IMU data, and use the consistency of the IMU and camera observations of the same rigid body motion as the motion constraint for parameter optimization. Step 5: Based on visual geometric constraints and motion constraints, construct an extrinsic parameter optimization model with the goal of minimizing reprojection error and IMU motion consistency error, and output the extrinsic parameter matrix to achieve automatic calibration of multi-sensor extrinsic parameters.

[0007] Preferably, in step 1, the calibration board is set at the entrance of the lawnmower robot on the front of the charging pile. After the lawnmower robot enters the calibration working mode, it runs to a position at a set distance from the charging pile to ensure that the RGB-D camera's field of view covers the calibration board.

[0008] Preferably, in step 2, the lawnmower robot performs a combination of translation and rotation, so that the RGB-D camera is aligned with the calibration plate from different perspectives during the movement, and acquires multiple sets of RGB images and depth information data.

[0009] Preferably, step 3 specifically includes: The corner pixel coordinates and their depth values ​​detected by the RGB-D camera are filtered using median filtering to eliminate holes and noise in the depth information. A local coordinate system for the calibration board is established, and coarse matching of feature points is performed based on the direct mapping relationship between the detected corner points and the local coordinate system of the calibration board. PnP is used to calculate the initial pose transformation and perform fine matching of feature points; Four pairs of points are randomly sampled to calculate candidate transformations. The number of matching points with projection errors less than the threshold is counted. The optimal model is selected iteratively, and mismatched points are eliminated. Finally, the set of observed coordinate points of the corner points in the camera coordinate system and the set of theoretical coordinate points of the corner points in the local coordinate system of the calibration board are obtained.

[0010] Preferably, in step 4, the IMU data is output in discrete form, and the median integration method is used to convert the continuous integral into discrete accumulation.

[0011] Preferably, the extrinsic parameter optimization model iteratively optimizes the objective function using the Levenberg-Marquardt algorithm.

[0012] As a preferred method, the automated calibration method for multi-sensor extrinsic parameters of lawn mowing robots measures the calibration accuracy by calculating the reprojection error. If the error exceeds the set error threshold, a second calibration is performed.

[0013] As a preferred embodiment, the automated calibration system for multi-sensor extrinsic parameters of lawnmowers is suitable for automated calibration methods of multi-sensor extrinsic parameters of lawnmowers, including: The intelligent lawn mowing robot module is equipped with an RGB-D camera, an inertial measurement unit, and a main control system. The main control system is embedded with a calibration program and is connected to the RGB-D camera and the inertial measurement unit to trigger the calibration mode, control the robot's movement, and perform data processing. The calibration plate is fixed at the entrance on the front of the charging pile to provide a calibration benchmark.

[0014] Preferably, the RGB-D camera includes: The corner detection unit is used to extract the corner points of the calibration board from the RGB image; The depth filtering unit uses median filtering to eliminate holes and noise in the depth map; The coordinate transformation unit calculates the theoretical coordinates of corner points based on the local coordinate system of the calibration plate and achieves fine matching of feature points through the PnP algorithm. The inertial measurement unit includes: The data sampling unit acquires the angular velocity and acceleration data of the IMU at a fixed frequency; The median integration unit converts continuous integration into discrete accumulation to calculate rotational and velocity increments.

[0015] Preferably, the calibration plate includes: The checkerboard calibration plate is aligned with the center of the RGB-D camera's field of view. An adjustable angle bracket is used to install the calibration plate plane perpendicular to the central axis of the charging pile entrance; A waterproof acrylic layer covers the surface of the calibration plate; Controllable LED light strip, integrated into the edge of the calibration board, is used for low-light supplemental lighting; Reflective markers are placed at the four corners of the calibration board to assist the lawnmower robot in initial positioning.

[0016] The beneficial effects of this invention are as follows: By setting up a calibration board, this invention achieves automated calibration of the extrinsic parameters of the RGB-D camera and IMU through the automatic movement process of the lawnmower robot when it is not performing work tasks. This invention greatly simplifies the calibration process, improves the accuracy and efficiency of calibration, and thus enhances the positioning and navigation capabilities of the lawnmower robot, meeting the needs of modern gardening management for intelligent robots. Attached Figure Description

[0017] Figure 1 This is a flowchart of the present invention; Figure 2 This is a schematic diagram of the reprojection error of the present invention. Detailed Implementation

[0018] Exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, these exemplary embodiments can be implemented in many forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided to make this application more comprehensive and complete, and to fully convey the concept of the exemplary embodiments to those skilled in the art.

[0019] Furthermore, the described features, structures, or characteristics can be combined in any suitable manner in one or more embodiments. Numerous specific details are provided in the following description to give a thorough understanding of embodiments of this application. However, those skilled in the art will recognize that the technical solutions of this application can be practiced without one or more of the specific details, or other methods, components, apparatuses, steps, etc., can be employed. In other instances, well-known methods, apparatuses, implementations, or operations are not shown or described in detail to avoid obscuring various aspects of this application.

[0020] The flowcharts shown in the accompanying drawings are merely illustrative and do not necessarily include all content and operations / steps, nor do they necessarily have to be performed in the described order. For example, some operations / steps can be broken down, while others can be combined or partially combined; therefore, the actual execution order may change depending on the specific circumstances.

[0021] Example: Automated calibration method for multi-sensor extrinsic parameters of lawnmower robots, such as Figure 1 As shown, it includes the following steps: Step 1: The lawnmower enters calibration mode, and the RGB-D camera's field of view covers the calibration board; Step 2: The lawnmower robot executes a preset motion sequence. During the motion, the RGB-D camera synchronously acquires RGB images and depth information, and the inertial measurement unit acquires IMU data. Step 3: The RGB-D camera detects the calibration board corner points in the RGB image, and calculates the 3D coordinates of the corner points in the camera coordinate system by combining the depth information. The reprojection consistency of the corner points observed by the camera in multiple frames of images is used as the visual geometric constraint for parameter optimization. Step 4: Calculate the relative motion between adjacent image frames using the angular velocity and acceleration of the acquired IMU data, and use the consistency of the IMU and camera observations of the same rigid body motion as the motion constraint for parameter optimization. Step 5: Based on visual geometric constraints and motion constraints, construct an extrinsic parameter optimization model with the goal of minimizing reprojection error and IMU motion consistency error, and output the extrinsic parameter matrix to achieve automatic calibration of multi-sensor extrinsic parameters.

[0022] In step 1, the calibration board is set at the entrance of the lawnmower robot on the front of the charging pile. After the lawnmower robot enters the calibration working mode, it runs to a position 0.3-0.5m away from the charging pile to ensure that the RGB-D camera's field of view covers the calibration board.

[0023] In step 2, the lawnmower robot performs a combination of translation and rotation, so that the RGB-D camera is aligned with the calibration plate from different perspectives during the movement, and acquires multiple sets of RGB images and depth information data.

[0024] Specifically, the lawnmower robot executes a preset motion sequence, controlling its movement speed to be less than 0.2 m / s and its angular velocity to be less than 30° / s. It moves left and right along a direction perpendicular to the charging station, within a range of ±0.5 m, keeping the calibration board within its field of view. The robot rotates ±30° around its central axis, simultaneously acquiring RGB images, depth information, and IMU data. A combination of translation and rotation ensures that the RGBD camera is aligned with the calibration board from different angles during the movement, collecting multiple sets of data.

[0025] Step 3 specifically includes: The corner pixel coordinates and their depth values ​​detected by the RGB-D camera are filtered using median filtering to eliminate holes and noise in the depth information. A local coordinate system for the calibration board is established, and coarse matching of feature points is performed based on the direct mapping relationship between the detected corner points and the local coordinate system of the calibration board. PnP is used to calculate the initial pose transformation and perform fine matching of feature points; Four pairs of points are randomly sampled to calculate candidate transformations. The number of matching points with projection errors less than the threshold is counted. The optimal model is selected iteratively, and mismatched points are eliminated. Finally, the set of observed coordinate points of the corner points in the camera coordinate system and the set of theoretical coordinate points of the corner points in the local coordinate system of the calibration board are obtained.

[0026] Specifically, the RGB-D camera detects corner points on the calibration board and calculates the 3D coordinates of these corner points in the camera coordinate system using depth information. The pixel coordinates (u, v) and depth value d of the corner points detected by the camera are processed using median filtering to eliminate holes and noise in the depth map. The 3D coordinates P of the corner points in the camera coordinate system are then calculated. c =[X c Y c Z c ] T for: Among them, f x f y c represents the camera's focal length in the x and y directions, respectively, reflecting the camera's image scaling ratio. x c y This represents the pixel coordinates of the camera's optical center on the image plane, i.e., the image center offset.

[0027] Establish a local coordinate system for the calibration board, with the center of the board surface as the origin and the Z-axis perpendicular to the board surface and pointing outwards. Assuming the checkerboard calibration board has dimensions of m×n and each square has a side length of s, the theoretical coordinates of the corner point in the i-th row and j-th column in the calibration board coordinate system are: Based on the direct mapping between detected corner points and theoretical coordinates, coarse feature point matching is performed. Then, PnP (Perspective-n-Point) is used to calculate the initial pose transformation, achieving fine feature point matching. Where R is the rotation matrix and t is the translation vector, used to convert the theoretical coordinates of the corner points. Transform to camera coordinate system The observed coordinates of the lower corner point in the camera coordinate system. Here are the theoretical coordinates of the lower corner point in the local coordinate system of the calibration plate, and ||·|| represents the Euclidean distance.

[0028] Four pairs of points are randomly sampled to calculate candidate transformations. The number of matching points with projection errors less than the threshold ∈ (∈ = 3.0 pixels) is counted. The optimal model is selected iteratively, and mismatched points are eliminated. Finally, the set of observed coordinate points of the corner points in the camera coordinate system is obtained. And the theoretical coordinate point set of the lower corner point in the local coordinate system of the calibration plate

[0029] In step 4, the IMU data is output in discrete form, and the median integration method is used to convert the continuous integral into discrete accumulation.

[0030] Specifically, in actual systems, the sampling frequency of IMU data is 100-500Hz, and the data is output in discrete form, requiring the continuous integration to be converted into discrete accumulation. The median integration method is used to improve accuracy. For the i-th IMU sample, the angular velocity is taken as the median of adjacent time points: Where b ω For the zero bias of the gyroscope, ω i ω i+1 The angular velocity measurements acquired by the IMU at the i-th and (i+1)-th times are respectively, and the rotation increment ΔR is given. i for: Where δ t For the IMU sampling interval, the overall pre-integration rotation is a successive left multiplication: Where, ΔR imu The overall rotation increment from time k to time N-1 is obtained by successively left-multiplying by the rotation increment matrix of each interval. Take the median value of the acceleration and transform it to the initial time coordinate system: Among them, a i a i+1 b represents the acceleration measurements acquired by the IMU at the i-th and (i+1)-th times, respectively. a To achieve zero bias in the accelerometer, Let be the median acceleration at time i, and let Δv be the velocity increment. imu for: Displacement increment Δp imu Calculated by double integral: Where, Δv i Let be the velocity increment for the i-th interval.

[0031] Finally, the IMU pre-integral quantity ΔR between adjacent time points is obtained. imu Δv imu and Δp imu Then, together with the visual reprojection error, an optimization problem is constructed, and the extrinsic parameter matrix is ​​solved through tight coupling.

[0032] Therefore, the objective function of the extrinsic parameter optimization model is: Where π(·) represents the camera projection model, Let be the pixel coordinates of the i-th feature point in the k-th frame. Let λ be the relative rotation of the camera calculated through feature point matching, λ be the weight coefficient, and log(·) be the Lie algebra mapping.

[0033] The extrinsic parameter optimization model described herein uses the Levenberg-Marquardt algorithm to iteratively optimize the objective function.

[0034] The automated calibration method for multi-sensor extrinsic parameters of lawnmower robots detects calibration accuracy by calculating reprojection error. If the error exceeds a set error threshold, a second calibration is performed. A specific diagram illustrating the reprojection error is shown below. Figure 2 As shown.

[0035] The specific process of secondary calibration is as follows: Assume a frame contains 100 feature points (N=100), and collect 5 sets of motion data (K=5); the average reprojection error is 2.5 pixels, and the IMU rotation residual is 5°.

[0036] First iteration: μ0 = 0.01, after the update the reprojection error is reduced to 1.8 pixels, accept the increment, μ1 = 0.001; Fifth iteration: the error is reduced to 0.8 pixels, the threshold is met, and the algorithm converges.

[0037] The final external parameters are a rotational error of 0.3° and a translational error of 3mm, which meet the design specifications.

[0038] An automated calibration system for multi-sensor extrinsic parameters of lawnmowers, applicable to automated calibration methods for multi-sensor extrinsic parameters of lawnmowers, including: The intelligent lawn mowing robot module is equipped with an RGB-D camera, an inertial measurement unit, and a main control system. The main control system is embedded with a calibration program and is connected to the RGB-D camera and the inertial measurement unit to trigger the calibration mode, control the robot's movement, and perform data processing. The calibration plate is fixed at the entrance on the front of the charging pile to provide a calibration benchmark.

[0039] The RGB-D camera includes: The corner detection unit is used to extract the corner points of the calibration board from the RGB image; The depth filtering unit uses median filtering to eliminate holes and noise in the depth map; The coordinate transformation unit calculates the theoretical coordinates of corner points based on the local coordinate system of the calibration plate and achieves fine matching of feature points through the PnP algorithm. The inertial measurement unit includes: The data sampling unit acquires the angular velocity and acceleration data of the IMU at a fixed frequency; The median integration unit converts continuous integration into discrete accumulation to calculate rotational and velocity increments.

[0040] The calibration plate includes: The checkerboard calibration plate is aligned with the center of the RGB-D camera's field of view. An adjustable angle bracket is used to install the calibration plate plane perpendicular to the central axis of the charging pile entrance; A waterproof acrylic layer covers the surface of the calibration plate; Controllable LED light strip, integrated into the edge of the calibration board, is used for low-light supplemental lighting; Reflective markers are placed at the four corners of the calibration board to assist the lawnmower robot in initial positioning.

[0041] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of the embodiments disclosed herein. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed herein.

[0042] It should be understood that this application is not limited to the precise structure described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of this application is limited only by the appended claims.

Claims

1. An automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot, characterized in that, Includes the following steps: Step 1: The lawnmower enters calibration mode, and the RGB-D camera's field of view covers the calibration board; Step 2: The lawnmower robot executes a preset motion sequence. During the motion, the RGB-D camera synchronously acquires RGB images and depth information, and the inertial measurement unit acquires IMU data. Step 3: The RGB-D camera detects the calibration board corner points in the RGB image, and calculates the 3D coordinates of the corner points in the camera coordinate system by combining the depth information. The reprojection consistency of the corner points observed by the camera in multiple frames of images is used as the visual geometric constraint for parameter optimization. Step 4: Calculate the relative motion between adjacent image frames using the angular velocity and acceleration of the acquired IMU data, and use the consistency of the IMU and camera observations of the same rigid body motion as the motion constraint for parameter optimization. Step 5: Based on visual geometric constraints and motion constraints, construct an extrinsic parameter optimization model with the goal of minimizing reprojection error and IMU motion consistency error, and output the extrinsic parameter matrix to achieve automatic calibration of multi-sensor extrinsic parameters.

2. The automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot according to claim 1, characterized in that, In step 1, the calibration board is set at the entrance of the lawnmower robot on the front of the charging pile. After the lawnmower robot enters the calibration working mode, it runs to a position at a set distance from the charging pile to ensure that the RGB-D camera's field of view covers the calibration board.

3. The automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot according to claim 1, characterized in that, In step 2, the lawnmower robot performs a combination of translation and rotation, so that the RGB-D camera is aligned with the calibration plate from different perspectives during the movement, and acquires multiple sets of RGB images and depth information data.

4. The automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot according to claim 1, characterized in that, Step 3 specifically includes: The corner pixel coordinates and their depth values ​​detected by the RGB-D camera are filtered using median filtering to eliminate holes and noise in the depth information. A local coordinate system for the calibration board is established, and coarse matching of feature points is performed based on the direct mapping relationship between the detected corner points and the local coordinate system of the calibration board. PnP is used to calculate the initial pose transformation and perform fine matching of feature points; Four pairs of points are randomly sampled to calculate candidate transformations. The number of matching points with projection errors less than the threshold is counted. The optimal model is selected iteratively, and mismatched points are eliminated. Finally, the set of observed coordinate points of the corner points in the camera coordinate system and the set of theoretical coordinate points of the corner points in the local coordinate system of the calibration board are obtained.

5. The automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot according to claim 1, characterized in that, In step 4, the IMU data is output in discrete form, and the median integration method is used to convert the continuous integral into discrete accumulation.

6. The automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot according to claim 1, characterized in that, The extrinsic parameter optimization model described herein uses the Levenberg-Marquardt algorithm to iteratively optimize the objective function.

7. The automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot according to claim 6, characterized in that, The reprojection error is also calculated. If the error exceeds the set error threshold, a second calibration is performed.

8. An automated calibration system for multi-sensor extrinsic parameters of a lawnmower robot, applicable to the automated calibration method for multi-sensor extrinsic parameters of a lawnmower robot as described in any one of claims 1-7, characterized in that, include: The intelligent lawn mowing robot module is equipped with an RGB-D camera, an inertial measurement unit, and a main control system. The main control system is embedded with a calibration program and is connected to the RGB-D camera and the inertial measurement unit to trigger the calibration mode, control the robot's movement, and perform data processing. The calibration plate is fixed at the entrance on the front of the charging pile to provide a calibration benchmark.

9. The automated calibration system for multi-sensor extrinsic parameters of a lawnmower robot according to claim 8, characterized in that, The RGB-D camera includes: The corner detection unit is used to extract the corner points of the calibration board from the RGB image; The depth filtering unit uses median filtering to eliminate holes and noise in the depth map; The coordinate transformation unit calculates the theoretical coordinates of corner points based on the local coordinate system of the calibration plate and achieves fine matching of feature points through the PnP algorithm. The inertial measurement unit includes: The data sampling unit acquires the angular velocity and acceleration data of the IMU at a fixed frequency; The median integration unit converts continuous integration into discrete accumulation to calculate rotational and velocity increments.

10. The automated calibration system for multi-sensor extrinsic parameters of a lawnmower robot according to claim 8, characterized in that, The calibration plate includes: The checkerboard calibration plate is aligned with the center of the RGB-D camera's field of view. An adjustable angle bracket is used to install the calibration plate plane perpendicular to the central axis of the charging pile entrance; A waterproof acrylic layer covers the surface of the calibration plate; Controllable LED light strip, integrated into the edge of the calibration board, is used for low-light supplemental lighting; Reflective markers are placed at the four corners of the calibration board to assist the lawnmower robot in initial positioning.

Images