A multi-sensor joint calibration method and system for an omnidirectional vehicle

Abstract

The application relates to a multi-sensor joint calibration method and system of an omnidirectional vehicle, wherein the method comprises the following steps: collecting and processing motion trajectories of various sensors on the omnidirectional vehicle in the process of performing a preset calibration motion in a preset calibration environment; calculating camera external parameter initial values of a camera sensor on the omnidirectional vehicle based on the motion trajectories; calculating radar external parameter initial values of a laser radar sensor on the omnidirectional vehicle based on the motion trajectories; calculating mechanical external parameters of a wheel odometer on the omnidirectional vehicle based on the motion trajectories; and constructing trajectory residuals based on the motion trajectories to jointly calibrate and optimize the camera external parameter initial values, the radar external parameter initial values and the mechanical external parameters, so as to obtain a multi-sensor joint calibration result of the omnidirectional vehicle. Through the application, the synchronous collection of multiple sensor data and chassis driving data is realized to estimate the external parameter initial values of the various sensors, and then the trajectory residuals are constructed to realize the joint calibration optimization with high precision, so that the flexibility and precision of the omnidirectional vehicle sensor calibration are effectively improved.

Description

Technical Field

[0001] This application relates to the field of robotics technology, and in particular to a multi-sensor joint calibration method and system for an omnidirectional vehicle. Background Technology

[0002] In the field of Autonomous Mobile Robots (AMRs), especially omnidirectional mobile robots (OMVs) with omnidirectional wheels, their high mobility—including in-situ rotation, lateral translation, and straight-line travel in any direction—makes them widely used in confined space operations such as warehousing and logistics, and precision manufacturing. Omnidirectional robots typically carry heterogeneous sensors such as wheel encoders, LiDAR, and RGB cameras. These sensors must undergo accurate extrinsic parameter calibration, namely, relative pose calibration between the sensors and calibration of the wheel encoder parameters, to unify the data into a single coordinate system.

[0003] Existing joint calibration schemes typically utilize pose data collected by various sensors on the AMR during the calibration process to jointly calibrate the extrinsic parameters of each sensor relative to the center of the AMR within a unified joint calibration framework. This method of joint calibration directly within a framework (such as nonlinear least squares method) can easily lead to over-coupling of the extrinsic parameter calibration between various sensors. That is, if the extrinsic parameter calibration of one sensor is incorrect, it is highly likely that other sensors will also malfunction.

[0004] Currently, no effective solution has been proposed for improving the ease of operation of joint calibration of multiple sensors for omnidirectional vehicles in related technologies. Summary of the Invention

[0005] This application provides a multi-sensor joint calibration method and system for omnidirectional vehicles, which at least solves the problem of how to improve the operational convenience of multi-sensor joint calibration of omnidirectional vehicles in related technologies.

[0006] In a first aspect, embodiments of this application provide a multi-sensor joint calibration method for an omnidirectional vehicle, the method comprising: During the execution of preset calibration motion in a preset calibration environment, the motion trajectories of various sensors on the omnidirectional vehicle are collected and processed. Based on the motion trajectory, the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle camera sensor are calculated; Based on the motion trajectory, the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle-mounted lidar sensor are calculated; Based on the motion trajectory, the mechanical external parameters of the omnidirectional vehicle wheel-mounted odometer are calculated; Based on the motion trajectory, a trajectory residual is constructed to jointly calibrate and optimize the initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters, thereby obtaining the multi-sensor joint calibration result of the omnidirectional vehicle.

[0007] In some embodiments, the acquisition and processing to obtain the motion trajectories of various sensors on the omnidirectional vehicle includes: For the image data of the omnidirectional vehicle during the preset calibration motion process collected by the camera sensor, the camera motion trajectory of the camera sensor is generated by the image recognition algorithm; For the point cloud data collected by the lidar sensor during the omnidirectional vehicle's execution of the preset calibration motion, the lidar sensor's radar motion trajectory is generated through a point cloud registration algorithm. The chassis drive data of the omnidirectional vehicle during the execution of the preset calibration motion is used to generate the chassis motion trajectory through the wheel odometer on the omnidirectional vehicle.

[0008] In some embodiments, the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle-mounted camera sensor are calculated based on the motion trajectory; Based on the camera motion trajectory, the initial values ​​of the camera extrinsic parameters and the yaw angle extrinsic parameters between the omnidirectional vehicle camera sensor and the chassis coordinate system are calculated using the least squares fitting algorithm. The initial values ​​of the camera extrinsic parameters include the initial values ​​of the rotational extrinsic parameters and the initial values ​​of the horizontal displacement extrinsic parameters.

[0009] In some embodiments, the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle-mounted lidar sensor are calculated based on the motion trajectory, including: Based on the radar motion trajectory, the initial values ​​of the radar extrinsic parameters between the omnidirectional vehicle-mounted lidar sensor and the chassis coordinate system are calculated using the least squares fitting algorithm and the circular trajectory fitting algorithm.

[0010] In some embodiments, the mechanical extrinsic parameters of the omnidirectional vehicle wheel-mounted odometer are calculated based on the motion trajectory, including: Based on the chassis motion trajectory, the mechanical external parameters of the wheeled odometer on the omnidirectional vehicle are calculated and determined using the omnidirectional vehicle forward kinematics formula. The coefficients to be optimized include the front steering wheel angle offset, the rear steering wheel angle offset, the front wheel diameter coefficient, the rear wheel diameter coefficient, and the axle length coefficient.

[0011] In some embodiments, constructing the trajectory residual based on the motion trajectory includes: With the yaw angle extrinsic parameter fixed, based on the chassis motion trajectory and the initial value of the camera extrinsic parameter, a first trajectory residual between the chassis motion trajectory and the camera sensor is constructed; Based on the chassis motion trajectory and the initial value of the radar extrinsic parameters, a second trajectory residual between the chassis motion trajectory and the lidar sensor is constructed.

[0012] In some embodiments, joint calibration and optimization are performed on the initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters to obtain the multi-sensor joint calibration results for the omnidirectional vehicle, including: Based on the first trajectory residual and the second trajectory residual, a global optimization objective function for minimizing the trajectory residual is constructed using Mahalanobis distance metric; The initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters are used as initial values ​​for joint optimization. Based on the global optimization objective function, joint calibration optimization is performed to obtain the multi-sensor joint calibration results of the omnidirectional vehicle.

[0013] In some embodiments, the setting of the preset calibration environment includes: A square area is enclosed by a baffle that is higher than the height of the omnidirectional vehicle laser; a QR code strip is affixed along the diagonal of the square area; a 5x5 QR code fixture is placed at the starting position of the QR code strip, parallel to the QR code strip, and the 5x5 QR code fixture is placed perpendicular to the ground.

[0014] In some embodiments, the preset calibration motion settings include: A side camera is magnetically fixed to the side of the omnidirectional vehicle; the omnidirectional vehicle to be calibrated is placed at the starting position of the QR code strip, with the forward direction aligned with the QR code strip, so that the bottom camera of the omnidirectional vehicle observes the QR code strip and the side camera observes the 5x5 QR code fixture. The omnidirectional vehicle is instructed to move in a straight line along the QR code strip and to rotate in place. During the linear driving motion and the stationary rotational motion, image data, point cloud data, and chassis drive data are collected respectively.

[0015] Secondly, embodiments of this application provide a multi-sensor joint calibration system for an omnidirectional vehicle. The system is used to execute the method described in the first aspect above. The system includes a data acquisition and processing module, a preliminary calibration module, and a joint calibration module. The acquisition and processing module is used to acquire and process the motion trajectories of various sensors on the omnidirectional vehicle during the execution of a preset calibration motion in a preset calibration environment. The preliminary calibration module is used to calculate the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle camera sensor based on the motion trajectory; calculate the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle lidar sensor based on the motion trajectory; and calculate the mechanical extrinsic parameters of the omnidirectional vehicle wheel odometer based on the motion trajectory. The joint calibration module is used to construct trajectory residuals based on the motion trajectory, so as to perform joint calibration optimization on the initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters, and obtain the multi-sensor joint calibration results of the omnidirectional vehicle.

[0016] Compared to related technologies, this application provides a multi-sensor joint calibration method and system for an omnidirectional vehicle. This method involves acquiring and processing the motion trajectories of various sensors on the omnidirectional vehicle during a preset calibration motion in a preset calibration environment. Based on the motion trajectories, initial values ​​of the camera extrinsic parameters of the camera sensor on the omnidirectional vehicle are calculated. Based on the motion trajectories, initial values ​​of the radar extrinsic parameters of the lidar sensor on the omnidirectional vehicle are calculated. Based on the motion trajectories, the mechanical extrinsic parameters of the wheeled odometer on the omnidirectional vehicle are calculated. Trajectory residuals are constructed based on the motion trajectories to jointly calibrate and optimize the initial values ​​of the camera extrinsic parameters, radar extrinsic parameters, and mechanical extrinsic parameters, resulting in a multi-sensor joint calibration result for the omnidirectional vehicle. This method achieves synchronous acquisition of data from multiple sensors and chassis drive data to estimate the initial values ​​of the extrinsic parameters of each sensor, and then constructs trajectory residuals to achieve high-precision global joint calibration optimization. This effectively improves the flexibility and accuracy of omnidirectional vehicle sensor calibration and solves the problem of how to improve the operational convenience of multi-sensor joint calibration for omnidirectional vehicles. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings: Figure 1 This is a flowchart of the steps of the omnidirectional vehicle multi-sensor joint calibration method according to an embodiment of this application; Figure 2 This is a schematic diagram of the calibration environment according to an embodiment of this application; Figure 3 This is a schematic diagram of some coordinate systems according to the embodiments of this application; Figure 4 This is a schematic diagram of the kinematic model of the central double steering wheel according to an embodiment of this application; Figure 5 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the application is described and illustrated below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application. All other embodiments obtained by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0019] Obviously, the accompanying drawings described below are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar scenarios based on these drawings without any inventive effort. Furthermore, it is understood that although the efforts made in this development process may be complex and lengthy, for those skilled in the art related to the content disclosed in this application, any changes to design, manufacturing, or production based on the technical content disclosed in this application are merely conventional technical means and should not be construed as insufficient disclosure of the content of this application.

[0020] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment that is mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0021] Unless otherwise defined, the technical or scientific terms used in this application shall have the ordinary meaning understood by one of ordinary skill in the art to which this application pertains. The terms “a,” “an,” “an,” “the,” and similar words used in this application do not indicate quantity limitation and may indicate singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or device that includes a series of steps or modules (units) is not limited to the listed steps or units, but may also include steps or units not listed, or may include other steps or units inherent to these processes, methods, products, or devices. The terms “connected,” “linked,” “coupled,” and similar words used in this application are not limited to physical or mechanical connections, but may include electrical connections, whether direct or indirect. “Multiple” used in this application refers to two or more. “And / or” describes the relationship between related objects, indicating that three relationships may exist; for example, “A and / or B” can represent: A alone, A and B simultaneously, and B alone. The character " / " generally indicates that the preceding and following objects are in an "or" relationship. The terms "first," "second," and "third" used in this application are merely to distinguish similar objects and do not represent a specific ordering of the objects.

[0022] This application provides a multi-sensor joint calibration method for an omnidirectional vehicle. Figure 1 This is a flowchart illustrating the steps of the omnidirectional vehicle multi-sensor joint calibration method according to an embodiment of this application, as follows: Figure 1 As shown, the method includes the following steps: Step S102: During the execution of the preset calibration motion in the preset calibration environment, the motion trajectories of each sensor on the omnidirectional vehicle are collected and processed. Step S102 specifically includes the following steps: Step S1021, setting the preset calibration environment includes: using a baffle higher than the height of the omnidirectional vehicle laser to enclose a square area; attaching a QR code strip along the diagonal of the square area; placing a 5x5 QR code fixture parallel to the QR code strip at the starting position of the QR code strip, with the 5x5 QR code fixture placed perpendicular to the ground. Step S1022, the preset calibration motion settings include: fixing a side camera to the side of the omnidirectional vehicle using a magnetic attraction method; placing the omnidirectional vehicle to be calibrated at the starting position of the QR code strip, aligning the forward direction with the QR code strip, so that the bottom camera of the omnidirectional vehicle observes the QR code strip, and the side camera observes the 5x5 QR code fixture; causing the omnidirectional vehicle to perform straight-line driving motion and stationary rotation motion along the QR code strip; during the straight-line driving motion and stationary rotation motion, respectively collecting image data, point cloud data and chassis drive data.

[0023] It should be noted that, Figure 2 This is a schematic diagram of the calibration environment according to an embodiment of this application, such as... Figure 2 As shown, on a horizontal ground, a square area (3m x 3m) is enclosed using wooden boards higher than the laser height of the omnidirectional vehicle. A QR code strip (2.4m long) is attached along the diagonal. A 5x5 QR code fixture, parallel to the QR code strip, is placed at the starting position of the QR code strip and perpendicular to the ground. A side camera is magnetically fixed to the side of the omnidirectional vehicle. Finally, the calibrated omnidirectional vehicle is placed at the starting position of the strip, with its forward direction aligned with the strip, so that the bottom camera of the omnidirectional vehicle can observe the bottom QR code strip and the side camera can observe the 5x5 QR code fixture. It should be further explained that the calibration motion is first performed by moving straight for 1.5m along the QR code strip at a speed of 0.05m / s, then moving back 1m, and finally rotating 360° in place at an angular velocity of 0.05rad / s. During the straight-line movement and the rotation, data is collected simultaneously, including point cloud data collected by the lidar sensor, image data of the calibration object collected by the camera sensor, and chassis drive data (such as wheel speed).

[0024] Step S1023: For the image data of the omnidirectional vehicle during the preset calibration motion process collected by the camera sensor, the camera motion trajectory of the camera sensor is generated by the image recognition algorithm; for the point cloud data of the omnidirectional vehicle during the preset calibration motion process collected by the lidar sensor, the lidar motion trajectory of the lidar sensor is generated by the point cloud registration algorithm; for the chassis drive data of the omnidirectional vehicle during the preset calibration motion process, the chassis motion trajectory is generated by the wheel odometer on the omnidirectional vehicle.

[0025] It should be noted that subsequent steps estimate the initial values ​​of the extrinsic parameters of each sensor based on the motion trajectories obtained in step S1023. This estimation of initial extrinsic parameters needs to be based on a coordinate system. Figure 3 This is a schematic diagram of some coordinate systems based on the embodiments of this application, such as... Figure 3 As shown, the origin of the chassis coordinate system (base coordinate system) of the omnidirectional vehicle (mobile robot) is located on the ground, with the X-axis pointing forward, the Y-axis pointing to the left, and the Z-axis perpendicular to the ground and pointing upward; the origin of the QR code coordinate system (tag coordinate system) is located at the upper left corner, with the X-axis pointing to the right, the Y-axis pointing towards the bottom of the QR code, and the Z-axis perpendicular to the QR code plane and pointing inward. By utilizing the prior knowledge of the mobile robot's straight-line and stationary rotational motions, the extrinsic parameters of the camera and LiDAR can be estimated in advance.

[0026] Step S104: Based on the motion trajectory, calculate the initial values ​​of the camera extrinsic parameters of the omnidirectional on-vehicle camera sensor; Specifically, in step S104, based on the camera motion trajectory, the initial values ​​of the camera extrinsic parameters and the yaw angle extrinsic parameters between the omnidirectional vehicle camera sensor and the chassis coordinate system are calculated using the least squares fitting algorithm. The initial values ​​of the camera extrinsic parameters include the initial values ​​of the rotational extrinsic parameters and the initial values ​​of the horizontal displacement extrinsic parameters.

[0027] Step S104 preferably involves estimating the extrinsic parameter pz (vertical distance from the camera to the strip) of the camera's vertical displacement based on the straight-line motion. (Initial values ​​of the rotational extrinsic parameters between the camera sensor and the chassis coordinate system). Specifically: First, let the trajectory pose of the camera during straight-line travel be represented as: , where R is the rotation transformation matrix (or Euler angles; 2D only has yaw angles, while 3D includes yaw angles, pitch angles, and roll angles), and t represents the displacement transformation. In 2D, t can be represented as {t(x),t(y)}, and in 3D it is {t(x),t(y),t(z)}.

[0028] Since the mobile robot moves on a horizontal surface, and the QR code tag coordinate system is also horizontally attached to the ground, the camera's extrinsic parameters... That is, the distance from the camera to the strip can be taken as the average of the z-values ​​corresponding to the displacement, then:

[0029] Where N is the number of trajectory poses. This is the value on the Z-axis.

[0030] Then, given that the mobile robot moves along the X-axis, the equation of the camera's trajectory line ax + by + c = 0 can be estimated using least squares:

[0031] in, and These represent the values ​​on the X and Y axes, respectively.

[0032] Then, by using the open-source library OpenCV to calculate the equation of the straight line, the rotational transformation heading angle from the chassis to the QR code strip can be obtained. Because the strip is parallel to the ground, i.e., the pitch angle... and roll angle Given (pitch is 0°, roll is 180°), the rotation transformation from the QR code strip coordinate system tag to the chassis coordinate system base can be calculated using Euler angles. :

[0033] Rotational pose during the straight-line movement of the mobile robot The values ​​remain roughly the same. The average rotation pose can be obtained by averaging the 3×3 rotation matrix and then performing SVD decomposition. Then the 3D rotation extrinsic parameters of the camera can be calculated:

[0034] Therefore, based on the mobile robot's linear motion along the X-axis, the camera extrinsic parameters pz and rotational extrinsic parameters can be estimated. By statistically analyzing the vertical displacement of the QR code strip by the camera during linear motion, the extrinsic parameters of the vertical displacement are calculated using the average value. Based on straight-line trajectory fitting (such as least squares method) and preset QR code plane pose (pitch=0°, roll=180°), the rotation deviation (yaw angle) between the camera and the chassis is calculated, reducing the initial angle error of the rotation extrinsic parameters and providing high-quality initial values ​​of extrinsic parameters for subsequent optimization.

[0035] Furthermore, based on the rotational trajectory of the camera sensor, the coordinates of the camera trajectory center are calculated using a circular trajectory fitting algorithm. Based on these center coordinates, the initial values ​​of the horizontal displacement extrinsic parameters px and py between the camera and the chassis coordinate system are then calculated. Specifically: First, during the rotation in place, let the motion trajectory pose of the camera, based on the pose of the first frame, be represented as follows: This corresponds to a circle centered at base. Let the center of this circle in the camera coordinate system be (m, n) and its radius be r. We can construct the least squares residual:

[0036] The center coordinates were then calculated using the open-source library OpenCV. The coordinates of the origin of the base coordinate system in the camera coordinate system of the first frame Then we have:

[0037]

[0038] in, This represents the coordinates of the origin of the base coordinate system within the camera (cam) coordinate system. Therefore, based on the in-situ rotational motion of the mobile robot, the previously estimated vertical displacement extrinsic parameter pz and the initial values ​​of the rotational extrinsic parameters are used. The initial values ​​of the camera's horizontal displacement extrinsic parameters px and py are estimated.

[0039] Furthermore, based on the deployment calibration environment, the yaw angle extrinsic parameter of the bottom camera relative to the omnidirectional vehicle center base can be obtained using the following formula:

[0040] in, This represents the angle from the side camera cam1 to the center base of the omnidirectional vehicle. Since the side camera is magnetically fixed to one side of the vehicle body, we can assume... The parameters are known. This indicates the angle of the side camera cam1 in the 5x5 QR code coordinate system tag1. This pose can be obtained by QR code sensing through the side camera. This represents the angle between the QR code strip coordinate system tag2 and the 5x5 QR code coordinate system tag1. Since deployment requires the QR code strip and the 5x5 QR code to be deployed parallel to each other, therefore... The parameters are known. This indicates the angle of the bottom camera cam2 in the QR code strip coordinate system tag2. This pose can be obtained by the bottom camera through QR code sensing. This indicates the angle of the bottom camera cam2 relative to the center base of the omnidirectional vehicle, which can be achieved through... Euler angles are calculated to obtain the yaw angle extrinsic parameters of the bottom camera relative to the center base of the omnidirectional vehicle.

[0041] Step S106: Based on the motion trajectory, calculate the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle-mounted lidar sensor; Specifically, in step S106, based on the radar motion trajectory, the initial values ​​of radar extrinsic parameters between the omnidirectional vehicle-mounted lidar sensor and the chassis coordinate system are calculated using the least squares fitting algorithm and the circular trajectory fitting algorithm.

[0042] Step S106 preferably involves calculating the initial values ​​of the extrinsic parameters between the lidar sensor and the chassis coordinate system using a least squares fitting and circular trajectory fitting algorithm. Specifically: Similar to the 3D extrinsic estimation of camera sensors, 2D extrinsic parameter estimation for lidar sensors involves first estimating the yaw angle of the laser extrinsic parameter through linear fitting, and then estimating it through circular trajectory fitting. , The displacement extrinsic parameters of the lidar sensor can be calculated as follows:

[0043] Extrinsic parameters of 2D lidar sensors can be obtained This refers to the initial values ​​of the radar extrinsic parameters of the lidar sensor.

[0044] Step S108: Based on the motion trajectory, calculate the mechanical external parameters of the omnidirectional vehicle wheel-type odometer; Specifically, in step S108, based on the chassis motion trajectory, the mechanical external parameters of the wheeled odometer on the omnidirectional vehicle are calculated and determined using the omnidirectional vehicle forward kinematics formula. The coefficients to be optimized include the front steering wheel angle offset, the rear steering wheel angle offset, the front wheel diameter coefficient, the rear wheel diameter coefficient, and the axle length coefficient.

[0045] Step S108 preferably involves the mechanical external parameters of the wheel-type odometer. Includes: front steering wheel angle offset Rear steering wheel angle offset Front wheel diameter coefficient Rear wheel diameter coefficient Axis length factor . Figure 4 This is a schematic diagram of the kinematic model of the central double steering wheel according to an embodiment of this application, as shown below. Figure 4 As shown, based on the omnidirectional vehicle forward kinematics formula combined with mechanical external parameters... The included coefficients to be optimized can be used to obtain the following chassis calibration model formula:

[0046]

[0047]

[0048] Where v0 and v1 are the front and rear wheel speeds of the chassis, respectively, calculated from encoder values ​​over a certain time interval:

[0049] in, Indicates a point in time At the appointed time The time interval, This indicates the change in encoder values ​​within the corresponding time interval. Indicates the total number of encoders. and These are the front and rear steering wheel angles currently issued by the chassis. This is the shaft length parameter. From this, the wheel angle offset, wheel diameter coefficient, and shaft length coefficient can be calculated and determined.

[0050] Step S110: Based on the motion trajectory, a trajectory residual is constructed to jointly calibrate and optimize the initial values ​​of camera extrinsic parameters, radar extrinsic parameters, and mechanical extrinsic parameters, thereby obtaining the multi-sensor joint calibration results of the omnidirectional vehicle.

[0051] Step S110 specifically includes the following steps: Step S1101: Fix the yaw angle extrinsic parameter, and based on the chassis motion trajectory and the initial value of the camera extrinsic parameter, construct the first trajectory residual between the chassis motion trajectory and the camera sensor; based on the chassis motion trajectory and the initial value of the radar extrinsic parameter, construct the second trajectory residual between the chassis motion trajectory and the lidar sensor.

[0052] Preferably, in step S1101, based on the mechanical extrinsic parameters of the wheel odometer, the initial value of the extrinsic parameters of the camera sensor (note that to avoid infinite solutions and failure to converge in the optimization process, the extrinsic parameters of the bottom camera yaw angle need to be fixed in the following optimization process), the initial value of the extrinsic parameters of the lidar sensor, and the chassis motion trajectory, the trajectory residuals between the chassis motion trajectory and the camera sensor, and the trajectory residuals between the chassis motion trajectory and the lidar sensor are constructed respectively.

[0053] Based on the external mechanical values ​​of wheel odometer The extrinsic parameters estimated by the camera and laser are used as initial values ​​for joint optimization. Linear interpolation is used to obtain the driving data, relative transformation poses of the camera and laser between time i and time j. The camera pose and extrinsic parameters are projected into a 2D form (unified with the observation pose dimension of the chassis and laser). The trajectory residuals of the wheel odometry, camera, and laser are constructed respectively:

[0054]

[0055] in This indicates the use of a wheel-based odometer calculation method, based on chassis drive data and mechanical external parameters. The pose obtained by motion modeling calculation [x] i , y i , θ i],, This represents the trajectory of the camera in the camera coordinate system. This represents the trajectory of the laser in the laser reference system. Indicates the extrinsic parameters of the camera sensor. This indicates the extrinsic parameters of the lidar sensor. Let se(2) be the Lie algebraic logarithmic mapping of 2D pose, where se(2) is in three-dimensional form (delta_x, delta_y, delta_theta).

[0056] It should be noted that by fixing the extrinsic parameters of the bottom camera's yaw angle in advance, and then combining the trajectory constraints from multiple independent sources (such as visual trajectory, laser matching trajectory, and wheel odometer) for joint optimization, we can avoid the situation where the optimization results in infinite solutions and cannot converge, and significantly reduce system drift and calibration errors.

[0057] Step S1102: Based on the first trajectory residual and the second trajectory residual, a global optimization objective function for minimizing the trajectory residual is constructed using Mahalanobis distance metric; the initial values ​​of camera extrinsic parameters, radar extrinsic parameters, and mechanical extrinsic parameters are used as initial values ​​for joint optimization, and joint calibration optimization is performed based on the global optimization objective function to obtain the multi-sensor joint calibration results of the omnidirectional vehicle.

[0058] Preferably, in step S1102, trajectory residuals are constructed in batches according to different time intervals, and weighted summation is performed according to the time intervals. A global optimization objective function is then constructed using Mahalanobis distance metric.

[0059] in: This represents the Mahalanobis distance (sum of squared residuals). Here, the covariance matrix is ​​used, with time intervals as weights. Therefore, based on the global optimization objective function, the joint calibration results of the mobile robot's multiple sensors are obtained through the trajectory residuals between the chassis trajectory and the camera sensor, and the trajectory residuals between the chassis trajectory and the LiDAR sensor.

[0060] Optionally, C++ code can be developed using the open-source Ceres Solver optimizer, and the Eigen library can be used to represent 2D and 3D poses to develop a wheeled odometer estimator. Template generics can be used to ensure that the odometer estimator supports automatic type conversion and is compatible with Ceres' automatic differentiation, avoiding manual differentiation. Then, only the chassis drive data, trajectory, and extrinsic parameters to be estimated need to be used to construct the residual function, and Ceres can be called to automatically complete the residual optimization. After the calculation is completed, the calibration results are compared with the design values ​​on the drawing. If there is no significant difference, the calibration is successful; otherwise, the calibration fails and needs to be recalibrated.

[0061] The method and steps provided in this application embodiment realize the synchronous acquisition of multiple sensor data and chassis drive data to estimate the initial values ​​of the external parameters of each sensor, and then construct the trajectory residual to achieve global high-precision joint calibration optimization, effectively improving the flexibility and accuracy of omnidirectional vehicle sensor calibration, and solving the problem of how to improve the operational convenience of multi-sensor joint calibration of omnidirectional vehicles.

[0062] It should be noted that the steps shown in the above process or in the flowchart of the accompanying figures can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0063] This application provides a multi-sensor joint calibration system for an omnidirectional vehicle. The system is used to execute the method provided in the above embodiments. The system includes a data acquisition and processing module, a preliminary calibration module, and a joint calibration module. The data acquisition and processing module is used to acquire and process the motion trajectories of various sensors on the omnidirectional vehicle during the execution of a preset calibration motion in a preset calibration environment. The preliminary calibration module is used to calculate the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle's camera sensor based on the motion trajectory; calculate the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle's lidar sensor based on the motion trajectory; and calculate the mechanical extrinsic parameters of the omnidirectional vehicle's wheeled odometer based on the motion trajectory. The joint calibration module is used to construct trajectory residuals based on the motion trajectory, so as to jointly calibrate and optimize the initial values ​​of camera extrinsic parameters, radar extrinsic parameters, and mechanical extrinsic parameters, and obtain the multi-sensor joint calibration results of the omnidirectional vehicle.

[0064] The system provided in this application embodiment realizes the synchronous acquisition of multiple sensor data and chassis drive data to estimate the initial values ​​of the external parameters of each sensor, and then constructs the trajectory residual to achieve global high-precision joint calibration optimization, effectively improving the flexibility and accuracy of omnidirectional vehicle sensor calibration, and solving the problem of how to improve the operational convenience of multi-sensor joint calibration of omnidirectional vehicles.

[0065] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0066] This embodiment provides an electronic device including a memory and a processor. The memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0067] Optionally, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0068] Optionally, the electronic device may further include a processor, memory, network interface, display screen, and input device connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a multi-sensor joint calibration method for an omnidirectional vehicle. The display screen may be a liquid crystal display (LCD) or an e-ink display. The input device may be a touch layer covering the display screen, buttons, a trackball, or a touchpad mounted on the device's casing, or an external keyboard, touchpad, or mouse.

[0069] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated here.

[0070] Furthermore, in conjunction with the multi-sensor joint calibration method for omnidirectional vehicles in the above embodiments, this application embodiment can provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the multi-sensor joint calibration methods for omnidirectional vehicles in the above embodiments.

[0071] In one embodiment, Figure 5 This is a schematic diagram of the internal structure of an electronic device according to an embodiment of this application, such as... Figure 5 As shown, an electronic device is provided, which can be a server, and its internal structure diagram can be as follows. Figure 5 As shown, the electronic device includes a processor, a network interface, internal memory, and non-volatile memory connected via an internal bus. The non-volatile memory stores the operating system, computer programs, and a database. The processor provides computing and control capabilities, the network interface communicates with external terminals via a network, the internal memory provides an environment for the operating system and computer programs to run, the computer programs are executed by the processor to implement a multi-sensor joint calibration method for an omnidirectional vehicle, and the database stores data.

[0072] Those skilled in the art will understand that Figure 5The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the electronic device to which the present application is applied. A specific electronic device may include more or fewer components than those shown in the figure, or may combine certain components, or may have different component arrangements.

[0073] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory may include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0074] Those skilled in the art should understand that the technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments have been described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0075] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A multi-sensor joint calibration method for an omnidirectional vehicle, characterized in that, The method includes: During the execution of preset calibration motion in a preset calibration environment, the motion trajectories of various sensors on the omnidirectional vehicle are collected and processed. Based on the motion trajectory, the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle camera sensor are calculated; Based on the motion trajectory, the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle-mounted lidar sensor are calculated; Based on the motion trajectory, the mechanical external parameters of the omnidirectional vehicle wheel-mounted odometer are calculated; Based on the motion trajectory, a trajectory residual is constructed to jointly calibrate and optimize the initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters, thereby obtaining the multi-sensor joint calibration result of the omnidirectional vehicle.

2. The method according to claim 1, characterized in that, The motion trajectories of various sensors on the omnidirectional vehicle obtained through data acquisition and processing include: For the image data of the omnidirectional vehicle during the preset calibration motion process collected by the camera sensor, the camera motion trajectory of the camera sensor is generated by the image recognition algorithm; For the point cloud data collected by the lidar sensor during the omnidirectional vehicle's execution of the preset calibration motion, the lidar sensor's radar motion trajectory is generated through a point cloud registration algorithm. The chassis drive data of the omnidirectional vehicle during the execution of the preset calibration motion is used to generate the chassis motion trajectory through the wheel odometer on the omnidirectional vehicle.

3. The method according to claim 2, characterized in that, Based on the motion trajectory, the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle camera sensor are calculated; Based on the camera motion trajectory, the initial values ​​of the camera extrinsic parameters and the yaw angle extrinsic parameters between the omnidirectional vehicle camera sensor and the chassis coordinate system are calculated using the least squares fitting algorithm. The initial values ​​of the camera extrinsic parameters include the initial values ​​of the rotational extrinsic parameters and the initial values ​​of the horizontal displacement extrinsic parameters.

4. The method according to claim 3, characterized in that, Based on the motion trajectory, the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle-mounted lidar sensor are calculated as follows: Based on the radar motion trajectory, the initial values ​​of radar extrinsic parameters between the omnidirectional vehicle-mounted lidar sensor and the chassis coordinate system are calculated using the least squares fitting algorithm and the circular trajectory fitting algorithm.

5. The method according to claim 4, characterized in that, Based on the motion trajectory, the mechanical external parameters of the omnidirectional vehicle wheel-mounted odometer are calculated as follows: Based on the chassis motion trajectory, the mechanical external parameters of the wheeled odometer on the omnidirectional vehicle are calculated and determined using the omnidirectional vehicle forward kinematics formula. The coefficients to be optimized include the front steering wheel angle offset, the rear steering wheel angle offset, the front wheel diameter coefficient, the rear wheel diameter coefficient, and the axle length coefficient.

6. The method according to claim 5, characterized in that, Constructing trajectory residuals based on the motion trajectory includes: With the yaw angle extrinsic parameter fixed, based on the chassis motion trajectory and the initial value of the camera extrinsic parameter, a first trajectory residual between the chassis motion trajectory and the camera sensor is constructed; Based on the chassis motion trajectory and the initial value of the radar extrinsic parameters, a second trajectory residual between the chassis motion trajectory and the lidar sensor is constructed.

7. The method according to claim 6, characterized in that, The initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters are jointly calibrated and optimized to obtain the multi-sensor joint calibration results of the omnidirectional vehicle, including: Based on the first trajectory residual and the second trajectory residual, a global optimization objective function for minimizing the trajectory residual is constructed using Mahalanobis distance metric; The initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters are used as initial values ​​for joint optimization. Based on the global optimization objective function, joint calibration optimization is performed to obtain the multi-sensor joint calibration results of the omnidirectional vehicle.

8. The method according to claim 1, characterized in that, The preset calibration environment settings include: A square area is enclosed by a baffle that is higher than the height of the omnidirectional vehicle laser; a QR code strip is affixed along the diagonal of the square area; a 5x5 QR code fixture is placed at the starting position of the QR code strip, parallel to the QR code strip, and the 5x5 QR code fixture is placed perpendicular to the ground.

9. The method according to claim 8, characterized in that, The preset calibration motion settings include: A side camera is magnetically fixed to the side of the omnidirectional vehicle; the omnidirectional vehicle to be calibrated is placed at the starting position of the QR code strip, with the forward direction aligned with the QR code strip, so that the bottom camera of the omnidirectional vehicle observes the QR code strip and the side camera observes the 5x5 QR code fixture. The omnidirectional vehicle is instructed to move in a straight line along the QR code strip and to rotate in place. During the linear driving motion and the stationary rotational motion, image data, point cloud data, and chassis drive data are collected respectively.

10. A multi-sensor joint calibration system for an omnidirectional vehicle, characterized in that, The system is used to perform the method according to any one of claims 1 to 9, and the system includes an acquisition and processing module, a preliminary calibration module, and a joint calibration module; The acquisition and processing module is used to acquire and process the motion trajectories of various sensors on the omnidirectional vehicle during the execution of a preset calibration motion in a preset calibration environment. The preliminary calibration module is used to calculate the initial values ​​of the camera extrinsic parameters of the omnidirectional vehicle camera sensor based on the motion trajectory; calculate the initial values ​​of the radar extrinsic parameters of the omnidirectional vehicle lidar sensor based on the motion trajectory; and calculate the mechanical extrinsic parameters of the omnidirectional vehicle wheel odometer based on the motion trajectory. The joint calibration module is used to construct trajectory residuals based on the motion trajectory, so as to perform joint calibration optimization on the initial values ​​of the camera extrinsic parameters, the initial values ​​of the radar extrinsic parameters, and the mechanical extrinsic parameters, and obtain the multi-sensor joint calibration results of the omnidirectional vehicle.

Images