An external parameter calibration method, device, apparatus and storage medium

Abstract

Embodiments of the present application disclose a kind of external parameter calibration method, device, equipment and storage medium.The method comprises: determining the first motion trajectory of camera according to image data, determining the second motion trajectory of wheeled odometry according to encoding value and preset wheeled odometry internal parameter;Get all adjacent key frames in the first motion trajectory, calculate the first rotation increment and the first translation increment between adjacent key frames;According to the acquisition time stamp of adjacent key frame, determine the corresponding chassis trajectory point in the second motion trajectory, and calculate the second rotation increment and the second translation increment between the corresponding chassis trajectory point;Based on hand-eye calibration equation set, according to the first rotation increment, the first translation increment, the second rotation increment and the second translation increment newly obtained, recursive calculation new internal parameter and external parameter, until internal parameter and external parameter converge.The above technical means, solve the technical problem that the external parameter calibration precision is low due to internal parameter change in prior art.

Description

Technical Field

[0001] This application relates to the field of multi-sensor fusion positioning technology, and in particular to an external parameter calibration method, apparatus, device and storage medium. Background Technology

[0002] State estimation based on multi-sensor fusion is a localization algorithm for autonomous driving. Multi-sensor fusion can compensate for the measurement accuracy problems of individual sensors. Sensor calibration is essential for the sensors to work together seamlessly. Vehicle-mounted cameras and wheeled odometers are common multi-sensor devices. The traditional calibration scheme for vehicle-mounted cameras and wheeled odometers involves constructing a hand-eye calibration problem about extrinsic parameters using the motion trajectories of the camera and the wheeled odometer to solve for the extrinsic parameters.

[0003] The inventors discovered that traditional calibration schemes use manually measured wheel odometer intrinsic parameters as fixed constants. However, in reality, the size of the left and right wheels is affected by tire pressure, thus influencing the intrinsic parameters. Changes in the intrinsic parameters lead to errors in trajectory extrapolation results, consequently affecting the accuracy of extrinsic parameter calibration. Secondly, traditional calibration schemes do not perform time calibration on the camera and wheel odometer. Without hardware synchronization between the camera and wheel odometer, the timestamp delay can be significant, affecting the accuracy of extrinsic parameter calibration. Traditional calibration schemes fit the camera's discrete trajectory into a continuous trajectory using image data, but the image data acquisition frequency is generally low, easily leading to low accuracy of the fitted curve, affecting the accuracy of extrinsic parameter calibration. Furthermore, traditional calibration schemes use wheeled mobile robots for calibration, but the motion degradation problem of the wheeled mobile robot is not considered during extrinsic parameter calibration, which also affects the accuracy of extrinsic parameter calibration. Summary of the Invention

[0004] This application provides an external parameter calibration method, apparatus, device, and storage medium, which solves the technical problem of low external parameter calibration accuracy caused by changes in internal parameters in the prior art.

[0005] In a first aspect, embodiments of this application provide an external parameter calibration method, including:

[0006] The system acquires image data captured by a camera and encoded values ​​acquired by a wheel odometer. Based on the image data, it determines the first motion trajectory of the camera. Based on the encoded values ​​and preset wheel odometer intrinsic parameters, it determines the second motion trajectory of the wheel odometer.

[0007] Obtain all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between the adjacent keyframes, and the first rotation increment or the first translation increment between the adjacent keyframes is greater than the corresponding preset increment threshold.

[0008] Based on the acquisition timestamps of the adjacent keyframes, determine the corresponding chassis trajectory points in the second motion trajectory, and calculate the second rotation increment and the second translation increment between the corresponding chassis trajectory points;

[0009] Based on the preset hand-eye calibration equations, new intrinsic and extrinsic parameters are recursively calculated according to the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment, until the intrinsic and extrinsic parameters converge.

[0010] In a second aspect, embodiments of this application provide an external parameter calibration device, comprising:

[0011] The motion trajectory determination module is configured to acquire image data collected by the camera and encoded values ​​collected by the wheel odometer, determine the first motion trajectory of the camera based on the image data, and determine the second motion trajectory of the wheel odometer based on the encoded values ​​and preset wheel odometer intrinsic parameters.

[0012] The camera increment determination module is configured to acquire all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between the adjacent keyframes, wherein the first rotation increment or the first translation increment between the adjacent keyframes is greater than the corresponding preset increment threshold.

[0013] The chassis increment determination module is configured to determine the corresponding chassis trajectory point in the second motion trajectory based on the acquisition timestamp of the adjacent key frames, and calculate the second rotation increment and the second translation increment between the corresponding chassis trajectory points.

[0014] The linear solution module is configured to recursively calculate new intrinsic and extrinsic parameters based on a preset set of hand-eye calibration equations, according to the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment, until the intrinsic and extrinsic parameters converge.

[0015] In a third aspect, embodiments of this application provide an external parameter calibration device, including: a processor, a memory, an input device, and an output device, wherein the processor, memory, input device, and output device are connected together, wherein:

[0016] The memory is used to store software programs, instructions, and modules;

[0017] The input device is used to receive input numeric or character information and send the numeric or character information to the processor;

[0018] The processor is configured to run software programs, instructions, and modules stored in the memory, implement the external parameter calibration method as described in the first aspect based on the digital or character information, and send the processing result to the output device.

[0019] The output device is used to output images or sounds based on the processing results.

[0020] In a fourth aspect, embodiments of this application provide a computer-readable storage medium having a computer program stored thereon that, when executed by a processor, implements the extrinsic parameter calibration method as described in the first aspect.

[0021] The aforementioned extrinsic parameter calibration method, apparatus, equipment, and storage medium solve for the extrinsic and intrinsic parameter variables in the hand-eye calibration equations using the first and second motion trajectories. Based on the new intrinsic parameters, the second motion trajectory of the chassis is recalculated, and this process is repeated recursively to calculate the intrinsic and extrinsic parameters until both converge. Since higher chassis pose accuracy leads to higher estimated extrinsic parameter accuracy, recursively updating the intrinsic parameters improves chassis pose accuracy, thereby enhancing extrinsic parameter accuracy. Attached Figure Description

[0022] Figure 1 This is a flowchart of an external parameter calibration method provided in one embodiment of this application;

[0023] Figure 2 This is a pose diagram illustrating the motion data acquisition process of the wheeled mobile robot provided in this embodiment;

[0024] Figure 3 This is a schematic diagram of the pose of the hand-eye calibration robot provided in an embodiment of this application;

[0025] Figure 4 This is a flowchart illustrating another external parameter calibration method provided in one embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the recursive calculation process provided in the embodiments of this application;

[0027] Figure 6 This is a flowchart illustrating another external parameter calibration method provided in one embodiment of this application;

[0028] Figure 7 This is a flowchart illustrating another external parameter calibration method provided in one embodiment of this application;

[0029] Figure 8 This is a schematic diagram of the structure of an external parameter calibration device provided in one embodiment of this application;

[0030] Figure 9 This is a schematic diagram of the structure of an external parameter calibration device provided in one embodiment of this application. Detailed Implementation

[0031] The present application will now be described in further detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and not for limiting the scope of the application. Furthermore, it should be noted that, for ease of description, only the parts relevant to the present application are shown in the drawings, not the entire structure.

[0032] It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity, operation, or object from another entity, operation, or object, and do not necessarily require or imply any such actual relationship or order between these entities, operations, or objects. For example, "first" and "second" in "first trajectory" and "second trajectory" are used to distinguish two different trajectories.

[0033] The extrinsic parameter calibration method provided in this application embodiment can be executed by an extrinsic parameter calibration device, which can be implemented by software and / or hardware. The extrinsic parameter calibration device can consist of two or more physical entities, or it can consist of a single physical entity. For example, the extrinsic parameter calibration device can be a smart device such as a computer.

[0034] The extrinsic parameter calibration device is equipped with at least one type of operating system, including but not limited to Android, Linux, and Windows. The extrinsic parameter calibration device may have at least one application installed on the operating system. This application may be a built-in application of the operating system or an application downloaded from a third-party device or server. In this embodiment, the extrinsic parameter calibration device has at least one application capable of executing the extrinsic parameter calibration method; therefore, the extrinsic parameter calibration device itself may also be the application.

[0035] For ease of understanding, the embodiments use a computer as an external parameter calibration device for exemplary description.

[0036] Figure 1 This is a flowchart illustrating an external parameter calibration method according to an embodiment of this application. (Reference) Figure 1 The external parameter calibration method includes:

[0037] S110. Acquire image data collected by the camera and encoded values ​​collected by the wheel odometer. Determine the first motion trajectory of the camera based on the image data. Determine the second motion trajectory of the wheel odometer based on the encoded values ​​and preset wheel odometer internal parameters.

[0038] The extrinsic parameter calibration process includes a motion data acquisition process and an extrinsic parameter estimation process. The motion data acquisition process involves controlling the wheeled mobile robot to move along a preset path. During this movement, the camera mounted on the wheeled mobile robot acquires image data including the chessboard calibration board at a certain frequency, and the wheeled odometer mounted on the chassis of the wheeled mobile robot acquires the encoded values ​​of the left and right wheels at a certain frequency. The extrinsic parameter estimation process involves calculating the rotational and translational transformation coordinates between the wheeled odometer coordinate system and the camera coordinate system based on the encoded values ​​of the left and right wheels acquired by the wheeled odometer and the image data acquired by the camera.

[0039] However, during the movement of a wheeled mobile robot, the size of the left and right wheels is affected by the tire pressure. At this time, the intrinsic parameters of the wheel odometer are no longer equal to the factory-set intrinsic parameters. If the factory-set intrinsic parameters are used to calculate the chassis pose accuracy, the estimated chassis pose accuracy will be low, resulting in low extrinsic parameter accuracy. Therefore, this embodiment considers the changes in the wheel odometer's intrinsic parameters when estimating extrinsic parameters to improve extrinsic parameter accuracy. The intrinsic parameters of the wheel odometer include the left wheel's internal parameters (r). l and right wheel r r The size of the wheel, and the baseline length b between the left and right wheels.

[0040] For example, the mobile data acquisition process includes a wheeled mobile robot moving along a preset path in a two-dimensional plane. During this movement, a camera and a wheeled odometer acquire image data and coded value data, respectively. The preset path can be a sinusoidal path or a figure-eight path. It should be noted that in traditional mobile data acquisition processes, the wheeled mobile robot moves along a purely rotational or purely translational path. This movement method does not consider the motion degradation problem of the wheeled robot, which affects the accuracy of extrinsic parameter calibration. Therefore, the sinusoidal and figure-eight movements performed by the wheeled mobile robot in this embodiment are non-degenerate movements, which can improve the accuracy of extrinsic parameter calibration. Furthermore, the wheeled mobile robot in this embodiment only moves within a two-dimensional plane. Except for the height Z, the other five degrees of freedom in the extrinsic parameters are theoretically observable, and the sinusoidal or figure-eight movements of the wheeled mobile robot ensure the observability of these five degrees of freedom.

[0041] Furthermore, after the mobile data acquisition process is completed, image data captured by the camera during the acquisition is obtained. This image data includes a chessboard calibration board image. Corner points of the chessboard calibration board in the image data are extracted using feature operators. When the number of corner points in an image exceeds eight, the PnP algorithm is used to estimate the camera's pose in the chessboard calibration board coordinate system when acquiring that image. After determining the pose of each image, the corresponding camera pose is fitted according to the chronological order of the image data acquisition timestamps to obtain the first motion trajectory.

[0042] Secondly, the left and right wheel coded value data collected by the wheel odometer during the data acquisition process are obtained. The left and right wheel coded value data are the increments of the left and right wheel coded values ​​between two adjacent acquisition timestamps. Assume that the chassis pose of the wheeled mobile robot at time k is T. k =[x k y k θ k ] T The angle Δφ of the left and right wheels is calculated based on the increment of the encoding values ​​of the left and right wheels between time k and time k+1. l and Δφ r The distance the left and right wheels traveled between time k and time k+1 was calculated using factory internal parameters. The chassis pose T at time k+1 was then calculated using the travel distance and the chassis pose at time k. k+1 The specific calculation process is performed using the following formula:

[0043] Δs l =r l Δφ l

[0044] Δs r =r r Δφ r

[0045]

[0046] Among them, T k+1 =[x k+1 y k+1 θ k+1 ] T Let Δs be the chassis pose at time k+1. l and Δs r These are the travel distances of the left and right wheels, r. l r r b and b represent the size of the left wheel, the size of the right wheel, and the baseline length between the left and right wheels, respectively.

[0047] After calculating the chassis pose corresponding to each acquisition timestamp, the chassis pose is fitted according to the order of acquisition timestamps from the wheel odometer to obtain the second motion trajectory.

[0048] Specifically, Figure 2 This is a pose diagram illustrating the motion data acquisition process of the wheeled mobile robot provided in this embodiment. For example... Figure 2 As shown, the coordinate system of the chessboard calibration plate is (X... w Y w Z w The camera coordinate system is (X). c Y c Z cThe wheel odometer coordinate system, also known as the chassis coordinate system, is (X... b Y b Z b In this context, the camera's world coordinate system is the chessboard calibration board coordinate system, and the wheeled odometer's world coordinate system represents the position of the wheeled mobile robot's chassis coordinate system at time zero. Both the chessboard calibration board's world coordinate system and the chassis's world coordinate system at time zero are geodetic coordinate systems, and their coordinates in the geodetic coordinate system can be predetermined. It can be understood that the camera pose is equivalent to the first extrinsic parameter matrix between the camera coordinate system and the chessboard calibration board coordinate system, the chassis pose is equivalent to the second extrinsic parameter matrix between the chassis coordinate system and the chassis coordinate system at time zero, and the third extrinsic parameter matrix between the chassis coordinate system at time zero and the chessboard calibration board coordinate system can be determined through their coordinates in the geodetic coordinate system. Therefore, by using the first, second, and third extrinsic parameter matrices, the camera coordinate system and the chassis coordinate system can be transformed to the same world coordinate system to estimate the extrinsic parameter matrix between them.

[0049] S120. Obtain all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between adjacent keyframes, and the first rotation increment or the first translation increment between adjacent keyframes is greater than the corresponding preset increment threshold.

[0050] As shown in step S110, each image data corresponds to a camera pose. The camera pose represents the position and orientation coordinates of the camera in the chessboard calibration board coordinate system when the image data is acquired. The rotation and translation increments between two camera poses can be considered as the rotation and translation transformation matrices between the corresponding two camera coordinate systems. If the rotation or translation transformation matrix between the camera poses of two adjacent image data sets is greater than a preset threshold transformation matrix, then these two image data sets are considered adjacent keyframes. For example, all image data pairs with adjacent acquisition timestamps in the first motion trajectory are acquired, and all adjacent keyframes are filtered out based on the transformation matrix between the image data pairs.

[0051] S130. Based on the acquisition timestamps of adjacent keyframes, determine the corresponding chassis trajectory points in the second motion trajectory, and calculate the second rotation increment and the second translation increment between the corresponding chassis trajectory points.

[0052] For example, the initial acquisition timestamps of the wheel odometer and the camera are the same, and the acquisition frequency of the wheel odometer is a multiple of the camera's acquisition frequency. While the camera is acquiring image data, the wheel odometer synchronously acquires coded value data. Therefore, the acquisition time of the camera pose in the first motion trajectory will be the same as the acquisition time of the chassis pose in the second motion trajectory. Thus, based on the acquisition timestamps of adjacent keyframes, two chassis poses with the same acquisition time are obtained from the second motion trajectory. Based on the position coordinates and attitude coordinates of these two chassis poses, the rotation transformation matrix and translation transformation matrix between the chassis poses are calculated, resulting in the second rotation increment and the second translation increment between the chassis trajectory points corresponding to adjacent keyframes.

[0053] S140. Based on the preset hand-eye calibration equation set, recursively calculate new intrinsic and extrinsic parameters according to the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment, until the intrinsic and extrinsic parameters converge.

[0054] For example, Figure 3 This is a schematic diagram of the pose of the hand-eye calibration robot provided in an embodiment of this application. Figure 3 As shown, the camera is mounted at the end effector of the robot arm. The purpose of robot hand-eye calibration is to determine the transformation matrix between the camera coordinate system C and the end effector coordinate system H. The transformation relationship between the end effector coordinate system H and the robot's base coordinate system B is B = A1H, where A1 is the transformation matrix between the end effector coordinate system H and the robot's base coordinate system B. The transformation relationship between the camera coordinate system C and the chessboard calibration board coordinate system G is G = A2C, where A2 is the transformation matrix between the camera coordinate system C and the chessboard calibration board coordinate system G. The transformation relationship between the camera coordinate system C and the end effector coordinate system H is H = A3C, where A3 is the transformation matrix used to determine the relationship between the camera coordinate system C and the end effector coordinate system H. From these three transformation relationships, we can obtain... Furthermore, the positions of the robot's base coordinate system B and the chessboard calibration board coordinate system G remain fixed, while the poses of the end effector coordinate system H and the camera coordinate system C change as the robot arm moves. Correspondingly, the transformation matrices A1, A2, and A3 also change. However, since the robot's base coordinate system B and the chessboard calibration board coordinate system G remain unchanged, the transformation relationship remains constant. In It will not change. Assume transformation matrices A1, A2, and A3 are the transformation matrices between the coordinate systems at the first time step, and transformation matrices A4, A5, and A6 are the transformation matrices between the coordinate systems at the second time step. At this time... Since the camera is fixedly mounted at the end of the robot arm, the transformation matrix between the camera coordinate system C and the end-effector coordinate system H remains constant, i.e., A3 = A5. Therefore, from... Push Treat A3 as the unknown variable X, Viewed as PX = XQ, hand-eye calibration then becomes a typical problem of solving for the unknown variable X in PX = XQ. Furthermore, according to the definition of a transformation matrix, X is a 4×4 homogeneous transformation matrix:

[0055]

[0056] Where R is the rotation transformation matrix in the transformation matrix, and t is the translation transformation matrix in the transformation matrix.

[0057] Furthermore, the transformation relationship between the end-effector coordinate system H and the robot base coordinate system B at the first moment is B = A1H1, and the transformation relationship between the end-effector coordinate system H and the robot base coordinate system B at the second moment is B = A4H2, where H1 and H2 are the poses of the end-effector coordinate system H at the first and second moments, respectively. From the above transformation relationships, we can obtain... Therefore, P is determined to be the transformation matrix between the end coordinate system at the first and second time points. Similarly, Q is the transformation matrix between the camera coordinate system at the first and second time points. Thus, by obtaining the transformation matrices of the camera coordinate system and the end coordinate system at different time points, and substituting them into PX = XQ, X can be obtained.

[0058] Understandable Figure 3 The camera coordinate system C in the image is equivalent to Figure 2 The camera coordinate system (X) c Y c Z c ), Figure 3 The coordinate system H at the end of the arm is equivalent to Figure 2 The chassis coordinate system (X) b Y b Z b ), Figure 3 The coordinate system G of the chessboard calibration plate is equivalent to Figure 2 The coordinate system of the chessboard calibration plate (X) w Y w Z w ), Figure 3 In this embodiment, the robot's base coordinate system B is equivalent to the chassis coordinate system at time zero. Therefore, this embodiment also solves for the camera coordinate system (X) based on hand-eye calibration. c Y c Z c ) and chassis coordinate system (X b Y b Z b The transformation matrix T between ) bc :

[0059]

[0060] Among them, Rbc Let t be the extrinsic rotation matrix. bc This is the extrinsic translation matrix. Furthermore, it represents the pose increment between adjacent keyframes acquired by the camera. It can be viewed as the transformation matrix Q, representing the pose increment between chassis trajectory points corresponding to adjacent keyframes. It can be viewed as a transformation matrix P, therefore Expanding this expression, we get:

[0061]

[0062] Multiplying the matrix further yields... as well as in, Know Let be the rotation transformation matrix and translation transformation matrix between the chassis trajectory points at time i and time j. Know Let be the rotation and translation transformation matrices between adjacent keyframes at time i and time j. Substitute the rotation and translation transformation matrices between all adjacent keyframes, as well as the rotation and translation transformation matrices between the corresponding chassis trajectory points of adjacent keyframes, into the hand-eye calibration equations. The transformation matrix T can then be obtained. bc .

[0063] However, as can be seen from step S130, the chassis pose T at time i is... bi =[x i-1 y i-1 θ i-1 ] T +[f(r l r r b), g(r) l r r b), q(r) l r r b)] T ,in It should be noted that the chassis position... The chassis pose at time i-1 is obtained by summing the chassis pose at time i-1 and the pose increment from time i-1 to time i. The pose increment from time i-1 to time i is obtained by using the wheel odometer intrinsic parameters and the left and right wheel encoding values ​​from time i-1 to time i. The chassis pose at time i-1 is obtained by summing the chassis pose at time i-2 and the pose increment from time i-2 to time i-1. Therefore, the chassis pose can be deduced. The chassis pose can be obtained by summing the chassis pose at time i-2, the pose increment from time i-1 to time i, and the pose increment from time i-2 to time i-1. Therefore, the chassis pose can be deduced. It is obtained by summing the chassis pose at time zero and the pose increments between adjacent times from time zero to time i, i.e. Where the chassis pose at time zero is [x0, y0, θ0]. T This refers to the coordinates of the chassis in the coordinate system at time zero. Similarly, the chassis pose at time j is... The chassis pose increment between time i and time j is obtained by summing the chassis pose at time 0 and the pose increments between adjacent times from time 0 to time j. It is obtained by summing the pose increments between adjacent time points from time i to time j. The pose increments between adjacent time points [f(r)] are then expressed as follows: l r r b), g(r) l r r b), q(r) l r r b)] T It can be viewed as a function of the intrinsic parameters of the wheel odometer, thus representing the chassis attitude increment. It can also be viewed as a function of the wheel odometer's intrinsic parameters, representing the translation transformation matrix between the chassis trajectory points at time i and time j. It is also a function with the intrinsic parameters of the wheel odometer as variables.

[0064] At this point, the hand-eye calibration equation is detailed. The variables in the calculation are intrinsic and extrinsic parameters. The chassis pose increment calculated from the factory intrinsic parameters is used as the initial value. Combined with the camera pose increment, the hand-eye calibration equations are solved to obtain new intrinsic and extrinsic parameters. Since the wheel odometer intrinsic parameters change during mobile data acquisition, a new second motion trajectory is derived based on the latest intrinsic parameters, resulting in a new chassis pose increment. The hand-eye calibration equations are then solved based on this new chassis pose increment to obtain new intrinsic and extrinsic parameters. If both the latest intrinsic and extrinsic parameters converge, they are used as the calibration intrinsic and extrinsic parameters. If either the latest intrinsic or extrinsic parameters fails to converge, new intrinsic and extrinsic parameters are recursively calculated based on the latest chassis pose increment and camera pose increment until they converge. Therefore, the specific steps S1401 to S1404 for recursively calculating the intrinsic and extrinsic parameters include:

[0065] S1401. Determine the camera's rotation axis vector based on the first rotation increment, align the rotation axis vector with the preset wheel odometer's rotation axis vector, and confirm the pitch and roll angles of the extrinsic rotation matrix.

[0066] For example, when a wheeled mobile robot rotates in a two-dimensional plane, the camera poses at different times can be transformed using a rotation transformation matrix. This matrix represents the rotational motion of the camera between two points. Therefore, the rotation transformation matrix between adjacent keyframes is obtained, and the unit vector of the camera's rotation axis is determined based on this matrix. It can be understood that both the rotation transformation matrix and the rotation vector can be used to represent the rotational motion between two points. The direction of the rotation vector indicates that the rotation axis is in the same direction, and the length (magnitude) of the rotation vector represents the angle (in radians) of counterclockwise rotation around the rotation axis. The rotation transformation matrix and the rotation vector can be converted to each other. Therefore, the unit vector of the rotation axis is obtained by normalizing the rotation vector.

[0067] Furthermore, since the wheeled mobile robot moves in a two-dimensional plane, the unit vector of the wheeled odometry's rotation axis is [0, 0, 1]. T Align the unit vector of the camera's rotation axis with the unit vector of the wheel odometer's rotation axis, and calculate the pitch and roll angles of the extrinsic rotation matrix.

[0068] S1402. Based on the latest obtained intrinsic parameters and encoding values, determine the new second motion trajectory. Based on the acquisition timestamps of adjacent keyframes, obtain the corresponding chassis trajectory points from the latest obtained second motion trajectory. Calculate the new second rotation increment and second translation increment based on the chassis trajectory points.

[0069] For example, after solving the hand-eye calibration equations to obtain new intrinsic and extrinsic parameters, if neither the intrinsic nor extrinsic parameters converge, a new second motion trajectory is regenerated based on the intrinsic parameters and coded values. However, the intrinsic parameters are independent of the camera's first motion trajectory, so the first motion trajectory remains unchanged, as do adjacent keyframes. Based on the acquisition timestamps of adjacent keyframes, the corresponding chassis pose is obtained from the latest obtained second motion trajectory, and the new chassis pose increment is calculated.

[0070] S1403. Substitute the newly obtained first rotation increment, first translation increment, second rotation increment, and second translation increment into the hand-eye calibration equation set to solve for the new intrinsic and extrinsic rotation matrix yaw angle and extrinsic translation matrix.

[0071] For example, based on the latest obtained chassis pose increment and the corresponding camera pose increment, the equations for hand-eye calibration are substituted into the equations to obtain the yaw angle and translation matrix of the new intrinsic and extrinsic rotation matrices.

[0072] S1404. If it is confirmed that the intrinsic and / or extrinsic parameters do not converge, the second motion trajectory is re-determined based on the latest obtained intrinsic and encoded data.

[0073] For example, if one of the yaw angle of the intrinsic and extrinsic rotation matrices or the extrinsic translation matrix fails to converge, the second motion trajectory is re-determined based on the latest obtained intrinsic and coded data, a new chassis pose increment is determined, and the new yaw angle of the intrinsic and extrinsic rotation matrices and the extrinsic translation matrix are obtained with the latest chassis pose increment until the intrinsic and extrinsic parameters converge simultaneously.

[0074] The aforementioned extrinsic parameter calibration method solves for the extrinsic and intrinsic parameters in the hand-eye calibration equations using the first and second motion trajectories. Based on the new intrinsic parameters, the second motion trajectory of the chassis is recalculated, and this process is repeated recursively to calculate both intrinsic and extrinsic parameters until they converge simultaneously. Since higher chassis pose accuracy leads to higher estimated extrinsic parameter accuracy, recursively updating the intrinsic parameters improves chassis pose accuracy, thereby enhancing extrinsic parameter accuracy.

[0075] Figure 4 This is a schematic flowchart illustrating another external parameter calibration method provided in one embodiment of this application. This embodiment is a specific modification based on the above embodiment. (Reference) Figure 4 The external parameter calibration method provided in this embodiment includes:

[0076] S210. Acquire image data collected by the camera and encoded values ​​collected by the wheel odometer. Determine the first motion trajectory of the camera based on the image data. Determine the second motion trajectory of the wheel odometer based on the encoded values ​​and preset wheel odometer internal parameters.

[0077] For example, image data captured by a camera during mobile data acquisition is obtained, and the camera pose corresponding to each acquisition timestamp is determined based on the image data. Encoded value data collected by a wheeled odometer during mobile data acquisition is also obtained, and the chassis pose corresponding to each acquisition timestamp is determined based on the encoded value data.

[0078] S220. Obtain all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between adjacent keyframes, and the first rotation increment or the first translation increment between adjacent keyframes is greater than the corresponding preset increment threshold.

[0079] For example, based on the camera pose increments of two adjacent acquisition timestamps, image data pairs with pose increments greater than a preset increment threshold are used as adjacent keyframes.

[0080] S230. Extract the corresponding first axial angle motion trajectory and second axial angle motion trajectory from the first motion trajectory and the second motion trajectory respectively, register the first axial angle motion trajectory and the second axial angle motion trajectory, and determine the acquisition delay time between the camera and the wheel odometer based on the timestamp of the paired trajectory points.

[0081] For example, when the camera and wheel odometer are not hardware synchronized, there is a time delay error between the camera's acquisition timestamp and the corresponding wheel odometer's acquisition timestamp. This delay error causes the camera pose and chassis pose for estimating extrinsic parameters to be at different times, resulting in low extrinsic parameter calibration accuracy. Therefore, this embodiment considers the acquisition delay time between the camera and wheel odometer to ensure that the camera pose and chassis pose for calculating extrinsic parameters are in the same spatiotemporal state, thereby improving the accuracy of extrinsic parameter calibration.

[0082] Specifically, the first motion trajectory consists of multiple camera poses, each including camera position and camera orientation. Changes in camera orientation are caused by camera rotation. Therefore, based on the camera orientation of each camera in the first motion trajectory, a first rotational motion trajectory for the camera can be generated. Rotation matrices and rotation vectors are mutually convertible. The axis angle is the angle of rotation around the rotation axis, and the length of the rotation vector represents the axis angle. The first rotational motion trajectory is converted into a corresponding first rotation vector, and the first axis angle motion trajectory for the camera is generated based on the length of the first rotation vector. Similarly, the second axis angle motion trajectory for the chassis can be extracted from the second motion trajectory of the chassis.

[0083] Furthermore, during the mobile data acquisition process, the rotation angles of the camera and the chassis are the same at the same moment. Therefore, the first and second axis angle motion trajectories are registered using ICP to ensure that the chassis pose and camera pose match at the same moment. After the chassis pose and camera pose are successfully paired, the difference between the acquisition timestamp of the paired chassis pose and the acquisition timestamp of the corresponding camera pose is used as the acquisition delay time between the camera and the wheel odometer.

[0084] S240. Based on the acquisition timestamps and delay times of adjacent keyframes, determine the acquisition timestamps of the chassis trajectory points corresponding to the adjacent keyframes, and obtain the corresponding chassis trajectory points from the second motion trajectory based on the acquisition timestamps of the chassis trajectory points.

[0085] Because there is a delay error between the camera's acquisition timestamp and the wheel odometer's acquisition timestamp, if the camera's pose acquisition timestamp is directly used as the chassis pose acquisition timestamp at the same moment to obtain the corresponding chassis pose, then the camera pose increment and the corresponding chassis pose increment do not occur in the same space-time, resulting in a large error in the external parameters calculated based on the camera pose increment and the chassis pose increment.

[0086] To address this, the acquisition timestamps and delay times of adjacent keyframes are added together to determine the encoded value data acquired synchronously with the adjacent keyframes, thereby determining the chassis pose synchronized with the camera pose.

[0087] S250. Based on the preset hand-eye calibration equation set, recursively calculate new intrinsic and extrinsic parameters according to the latest obtained first rotation increment, first translation increment, second rotation increment and second translation increment, until the intrinsic and extrinsic parameters converge.

[0088] It is understandable that the delay time is obtained based on the second motion trajectory. When the intrinsic parameters change, the second motion trajectory needs to be re-derived based on the new intrinsic parameters, which will also affect the delay time. Therefore, this embodiment considers the influence of the intrinsic parameters on the delay time and the second motion trajectory, and recursively calculates the new intrinsic and extrinsic parameters multiple times through inner and outer loops until the intrinsic and extrinsic parameters of the outer loop converge simultaneously. Accordingly, the recursive calculation steps of the inner and outer loops specifically include S2501 to S2503:

[0089] S2501. Calculate new intrinsic and extrinsic parameters through the first loop until the intrinsic and extrinsic parameters obtained in the first loop converge. The first loop includes updating the second motion trajectory based on the latest obtained intrinsic parameters, updating the second rotation increment and the second translation increment based on the new second motion trajectory, and updating the intrinsic and extrinsic parameters based on the new second rotation increment and the second translation increment.

[0090] For example, Figure 5 This is a schematic diagram of the recursive calculation process provided in an embodiment of this application. For example... Figure 5 As shown, the first loop, also known as the inner loop, includes: calculating a new second motion trajectory based on the latest obtained intrinsic parameters and encoding values; selecting a new chassis pose from the latest obtained second motion trajectories based on the acquisition timestamps of adjacent keyframes and the latest obtained delay time; calculating the chassis pose increment based on the latest obtained chassis pose; and solving the intrinsic and extrinsic parameters in the hand-eye calibration equations based on the latest obtained chassis pose increment and camera pose increment. The loop then determines whether the intrinsic and extrinsic parameters obtained in this round of calculation converge simultaneously. If they do not converge simultaneously, the second motion trajectory is redefined based on the latest obtained intrinsic parameters. If they converge simultaneously, the loop exits and enters the outer loop, i.e., the second loop.

[0091] S2502. When the intrinsic and extrinsic parameters obtained in the first loop converge, new intrinsic and extrinsic parameters are calculated through the second loop until the intrinsic and extrinsic parameters obtained in the second loop converge. The second loop includes updating the second motion trajectory based on the latest obtained intrinsic parameters, updating the delay time based on the new second motion trajectory, and updating the intrinsic and extrinsic parameters through the first loop.

[0092] like Figure 5As shown, the outer loop includes: receiving the latest output intrinsic and extrinsic parameters from the inner loop, and obtaining the received output intrinsic and extrinsic parameters from the inner loop in previous rounds. If the received intrinsic and extrinsic parameters converge simultaneously, and if they do not converge simultaneously, the second motion trajectory is updated according to the latest obtained intrinsic parameters. The second axis angle motion trajectory of the second motion trajectory is re-registered with the first axis angle motion trajectory to obtain a new delay time, and then the inner loop is entered. If they converge simultaneously, the intrinsic and extrinsic parameters are output, and the outer loop is exited.

[0093] S2503. When the intrinsic and extrinsic parameters obtained in the second loop converge, calculate the delay time using the latest obtained intrinsic parameters.

[0094] For example, the intrinsic parameters output by the outer loop can be regarded as the optimal intrinsic parameters. Therefore, the second motion trajectory is re-determined based on the intrinsic parameters, and a new delay time is calculated through the second motion trajectory. The delay time calculated at this time can be regarded as the optimal delay time.

[0095] The aforementioned extrinsic parameter calibration method registers the first axis angle motion trajectory and the second axis angle motion trajectory to determine the delay time between the camera's acquisition timestamp and the wheel odometer's acquisition timestamp. Based on the delay time, it corrects the chassis trajectory points corresponding to adjacent keyframes to ensure that the camera pose increment and chassis pose increment occur in the same spacetime. This, in turn, ensures higher accuracy of the intrinsic and extrinsic parameters obtained by solving the hand-eye calibration equations using the camera pose increment and chassis pose increment.

[0096] Figure 6 This is a schematic flowchart illustrating another external parameter calibration method provided in one embodiment of this application. This embodiment is a specific modification based on the above embodiment. (Reference) Figure 6 The external parameter calibration method provided in this embodiment includes:

[0097] S310: Acquire image data collected by the camera and encoded values ​​collected by the wheel odometer; determine the first motion trajectory of the camera based on the image data; and determine the second motion trajectory of the wheel odometer based on the encoded values ​​and preset wheel odometer internal parameters.

[0098] S320. Obtain all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between adjacent keyframes, and the first rotation increment or the first translation increment between adjacent keyframes is greater than the corresponding preset increment threshold.

[0099] S330. Based on the acquisition timestamps of adjacent keyframes, determine the corresponding chassis trajectory points in the second motion trajectory, and calculate the second rotation increment and the second translation increment between the corresponding chassis trajectory points.

[0100] S340. Based on the preset hand-eye calibration equation set, recursively calculate new intrinsic and extrinsic parameters according to the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment, until the intrinsic and extrinsic parameters converge.

[0101] For example, steps S310 to S340 can be specifically referred to as steps S110 to S140.

[0102] S350: Acquire the motion data collected by the inertial measurement unit, and fit the motion data and image data to form a third motion trajectory. The camera trajectory points of the third motion trajectory include the velocity of the camera coordinate system and the acquisition timestamp.

[0103] As can be seen from the above, the above embodiments fit the discrete trajectory of the camera into a continuous trajectory using image data. However, the acquisition frequency of image data is generally low, which can easily lead to low accuracy of the fitted curve and affect the accuracy of the extrinsic parameter calibration. Therefore, this embodiment further optimizes the intrinsic and extrinsic parameters through a nonlinear optimization process based on the intrinsic and extrinsic parameters finally obtained in the above embodiments.

[0104] Specifically, the wheeled mobile robot is equipped with an IMU (Installation Unit). During the data acquisition process, the IMU measures IMU data, and a continuous third motion trajectory is fitted using a B-spline algorithm to the image data and IMU data. The IMU data includes the camera's motion acceleration and velocity data. By arbitrarily obtaining any acquisition timestamp within the time range of the third motion trajectory, the corresponding camera pose and camera velocity can be interpolated. The camera velocity includes the angular velocity ω of the camera coordinate system. c and linear velocity v c .

[0105] S360. Determine the speed of the chassis coordinate system of the wheel odometer corresponding to the previous acquisition timestamp based on the incremental encoding value of the adjacent acquisition timestamp and the intrinsic parameter variables.

[0106] For example, the left wheel angular velocity ω of the chassis between time k and time k+1 is calculated based on the increment of the encoded value between time k and time k+1. l and the angular velocity ω of the right wheel r Thus, the linear velocity v of the left wheel can be derived. l =ω l r l The linear velocity of the right wheel is v r =ω r r r .

[0107] Furthermore, after calculating the linear velocities of the left and right wheels, the linear velocity of the chassis coordinate system in the x-axis direction at time k is derived based on the differential wheel motion model. and angular velocity in the z-axis direction The motion model of the differential wheel is shown below:

[0108]

[0109]

[0110] Where, r l and r r These are intrinsic parameters, which need to be solved through subsequent nonlinear optimization. Nonlinear optimization requires initial values ​​for these intrinsic parameters, and the convergent intrinsic parameters obtained by linearly solving the hand-eye calibration equation are the initial values ​​for these intrinsic parameters. Substituting these initial values ​​into the linear velocity calculation formula and the differential wheel motion model allows us to calculate the initial values ​​of the linear velocity and angular velocity in the chassis coordinate system.

[0111] It should be noted that, since the wheeled mobile robot moves in a two-dimensional plane, the linear velocity along the y-axis and z-axis of the chassis coordinate system is approximately zero, and the angular velocity along the x-axis and y-axis of the chassis coordinate system is approximately zero.

[0112] S370. Based on the timestamp of the encoded value, determine the corresponding camera trajectory point in the third motion trajectory, and obtain the velocity of the camera coordinate system accordingly.

[0113] For example, based on the acquisition timestamp of each encoded value data, the corresponding camera trajectory points are obtained from the third motion trajectory to determine the chassis pose and camera pose occurring in the same time and space, thereby determining the angular velocity ω relative to the chassis coordinate system. b The angular velocity ω in the corresponding camera coordinate system c The linear velocity v in the chassis coordinate system b The linear velocity v in the corresponding camera coordinate system c .

[0114] S380. Using the adjoint matrix of the external parameter variables, the velocity of the camera coordinate system is transformed into the chassis coordinate system, and the first error equation is constructed by combining it with the velocity of the chassis coordinate system corresponding to the acquisition timestamp.

[0115] The camera and wheeled odometer are fixedly mounted on the wheeled mobile robot. Therefore, the camera coordinate system and the chassis coordinate system lie on the same rigid body. The velocities between the two coordinate systems on the same rigid body are transformed using the adjoint matrix of the extrinsic parameter transformation matrix. By using the adjoint matrix of the extrinsic parameter variables, the velocity in the camera coordinate system is transformed into the chassis coordinate system, resulting in an estimated velocity value in the chassis coordinate system. The error between the estimated velocity value and the actual measured value is calculated; the smaller the error, the higher the calibration accuracy of the extrinsic parameter matrix.

[0116] Specifically, external parameters Its adjoint matrix Based on the adjoint matrix Tbc * The linear velocity v in the camera coordinate system c and angular velocity ω c Transform to the chassis coordinate system and compare it with the linear velocity v of the chassis coordinate system. b and angular velocity ω b By taking the difference, we obtain the first error equation:

[0117]

[0118] Because the first error equation includes the linear velocity v of the camera coordinate system c and angular velocity ω c Different first error equations can be constructed using camera speeds at different acquisition timestamps. Therefore, multiple first error equations can be constructed using speed data from multiple cameras to provide sufficient constraints on intrinsic and extrinsic variables.

[0119] It should be noted that solving the first error equation requires initial values ​​for the velocity estimate, which are obtained from the adjoint matrix of the extrinsic parameters and the linear velocity v in the camera coordinate system. c and angular velocity ω c By using the convergent extrinsic parameters obtained from the linear solution of the hand-eye calibration equation as the initial values ​​of the extrinsic parameters, the initial values ​​of the velocity estimate can be obtained.

[0120] S390. Use the newly obtained intrinsic and extrinsic parameters as the initial values ​​of the intrinsic and extrinsic parameters in the first error equation, and iteratively calculate the new intrinsic and extrinsic parameters using the Levenberg-Marquadt rule until the intrinsic and extrinsic parameters converge.

[0121] For example, the optimal intrinsic and extrinsic parameters in the first error equation can be determined by a nonlinear optimization algorithm. In order to suppress the singularity problem of the Hessian matrix, this embodiment uses an optimization algorithm with a trust region to iteratively optimize the intrinsic and extrinsic parameters until the intrinsic and extrinsic parameters converge simultaneously.

[0122] The aforementioned extrinsic parameter calibration method constructs a first error equation based on the motion velocities of the chassis coordinate system and the camera coordinate system. This first error equation is then used to further optimize the intrinsic and extrinsic parameters, improving the calibration accuracy. Furthermore, an optimization algorithm with a trust region is employed to avoid the singularity problem of the Hessian matrix, thus enhancing the optimization effect.

[0123] Figure 7 This is a schematic flowchart illustrating another external parameter calibration method provided in one embodiment of this application. This embodiment is a specific modification based on the above embodiment. (Reference) Figure 7 The external parameter calibration method provided in this embodiment includes:

[0124] S410: Acquire image data collected by the camera and encoded values ​​collected by the wheel odometer; determine the first motion trajectory of the camera based on the image data; and determine the second motion trajectory of the wheel odometer based on the encoded values ​​and preset wheel odometer internal parameters.

[0125] S420. Obtain all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between adjacent keyframes, and the first rotation increment or the first translation increment between adjacent keyframes is greater than the corresponding preset increment threshold.

[0126] S430. Extract the corresponding first axial angle motion trajectory and second axial angle motion trajectory from the first motion trajectory and the second motion trajectory respectively, register the first axial angle motion trajectory and the second axial angle motion trajectory, and determine the acquisition delay time between the camera and the wheel odometer based on the timestamp of the paired trajectory points.

[0127] S440. Based on the acquisition timestamps and delay times of adjacent keyframes, determine the acquisition timestamps of the chassis trajectory points corresponding to the adjacent keyframes, and obtain the corresponding chassis trajectory points from the second motion trajectory based on the acquisition timestamps of the chassis trajectory points.

[0128] S450: Based on the preset hand-eye calibration equation set, recursively calculate new intrinsic and extrinsic parameters according to the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment, until the intrinsic and extrinsic parameters converge.

[0129] For example, steps S410 to S450 can be specifically referred to as steps S210 to S250.

[0130] S460. Acquire the motion data collected by the inertial measurement unit, and fit the motion data and image data to form a third motion trajectory. The camera trajectory points of the third motion trajectory include the velocity, acceleration and acquisition timestamp of the camera coordinate system.

[0131] For example, IMU data is acquired, and a continuous third motion trajectory is fitted to the image data and IMU data using the B-spline algorithm. Any acquisition timestamp within the time range of the third motion trajectory can be interpolated to obtain the corresponding camera pose and the angular velocity ω of the camera coordinate system. c and linear velocity v c Angular acceleration of the camera coordinate system and linear acceleration

[0132] S470. Determine the speed of the chassis coordinate system of the wheel odometer corresponding to the previous acquisition timestamp based on the incremental encoding value of adjacent acquisition timestamps and the intrinsic parameter variables.

[0133] For example, the left wheel angular velocity ω of the chassis between time k and time k+1 is calculated based on the increment of the encoded value between time k and time k+1. l and the angular velocity ω of the right wheel r Then, based on the differential wheel motion model and intrinsic parameters, the angular velocity ω of the chassis coordinate system at time k is derived. b and linear velocity v b .

[0134] S480. Based on the acquisition timestamp of the encoded value, determine the corresponding camera trajectory point in the third motion trajectory, and obtain the velocity and acceleration of the camera coordinate system accordingly.

[0135] For example, based on the angular velocity ω of the chassis coordinate system b and linear velocity v b The acquisition timestamp is used to determine the angular velocity ω of the camera coordinate system corresponding to the same acquisition timestamp. c and linear velocity v c angular acceleration and linear acceleration

[0136] S490. Multiply the acceleration of the camera coordinate system by the increment of the delay time variable, and add the velocity of the camera coordinate system to obtain the delay compensation velocity of the camera coordinate system.

[0137] Because there is a time delay between the camera and the wheel odometer, the motion speed of the chassis coordinate system and the motion speed of the camera coordinate system corresponding to the same acquisition timestamp are not in the same space-time. Therefore, it is necessary to compensate for the camera's motion speed based on the time delay and the camera's motion acceleration in order to improve the calibration accuracy.

[0138] Specifically, the angular acceleration of the camera coordinate system and linear acceleration Simultaneously multiply by the derivative of the time delay variable δt, and then add the corresponding angular velocity ω of the camera coordinate system. c and linear velocity v c This yields the delay compensation speed of the camera coordinate system. The initial value of the delay time variable is the delay time obtained through the convergent intrinsic parameters output by the outer loop. It can be understood that the delay time is determined by the intrinsic parameters; when the intrinsic parameters change, the delay time also changes. Therefore, when optimizing the intrinsic parameters, the delay time needs to be optimized simultaneously, hence it is also set as a variable.

[0139] S500: By using the adjoint matrix of the external parameter variables, the delay compensation velocity of the camera coordinate system is transformed into the chassis coordinate system, and a second error equation is constructed with the velocity of the chassis coordinate system corresponding to the acquisition timestamp.

[0140] Specifically, the second error equation is:

[0141]

[0142] S510. Use the newly obtained intrinsic parameters, extrinsic parameters, and delay time as the initial values ​​of the intrinsic parameter variables, extrinsic parameter variables, and delay time variables in the second error equation. Iterate the intrinsic parameters and extrinsic parameters using the Levenberg-Marquadt rule until the intrinsic parameters, extrinsic parameters, and delay time converge.

[0143] For example, the intrinsic parameters, extrinsic parameters, and delay time in the second error equation are iteratively optimized using an optimization algorithm with a trust region until the intrinsic parameters, extrinsic parameters, and delay time converge simultaneously.

[0144] The above-mentioned external parameter calibration method constructs a second error equation by considering the motion speed of the chassis coordinate system, the motion speed of the camera coordinate system, the motion acceleration of the camera coordinate system, and the increment of the delay time. The second error equation is then used to further optimize the intrinsic parameters, external parameters, and delay time, thereby improving the calibration accuracy of the external parameters.

[0145] Figure 8 This is a schematic diagram of the structure of an external parameter calibration device provided in one embodiment of this application. (Reference) Figure 8 The external parameter calibration device includes: a motion trajectory determination module 601, a camera increment determination module 602, a chassis increment determination module 603, and a linear solution module 604.

[0146] The motion trajectory determination module is configured to acquire image data collected by the camera and encoded values ​​collected by the wheel odometer, determine the first motion trajectory of the camera based on the image data, and determine the second motion trajectory of the wheel odometer based on the encoded values ​​and preset wheel odometer internal parameters.

[0147] The camera increment determination module is configured to acquire all adjacent keyframes in the first motion trajectory, calculate the first rotation increment and the first translation increment between adjacent keyframes, and the first rotation increment or the first translation increment between adjacent keyframes is greater than the corresponding preset increment threshold.

[0148] The chassis increment determination module is configured to determine the corresponding chassis trajectory points in the second motion trajectory based on the acquisition timestamps of adjacent keyframes, and calculate the second rotation increment and the second translation increment between the corresponding chassis trajectory points.

[0149] The linear solution module is configured to recursively calculate new intrinsic and extrinsic parameters based on the preset hand-eye calibration equations, according to the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment, until the intrinsic and extrinsic parameters converge.

[0150] Based on the above embodiments, the linear solution module includes: a pitch and roll angle calculation unit, configured to determine the camera's rotation axis vector based on the first rotation increment, align the rotation axis vector with the preset rotation axis vector of the wheel odometer, and determine the pitch and roll angles of the extrinsic rotation matrix; a chassis pose increment update unit, configured to determine a new second motion trajectory based on the latest obtained intrinsic and encoded value data, obtain the corresponding chassis trajectory points from the latest obtained second motion trajectory based on the acquisition timestamps of adjacent keyframes, and calculate the new second rotation increment and second translation increment based on the chassis trajectory points; an intrinsic and extrinsic parameter update unit, configured to substitute the latest obtained first rotation increment, first translation increment, second rotation increment, and second translation increment into the hand-eye calibration equation system, and solve for the yaw angle and extrinsic translation matrix of the new intrinsic and new extrinsic rotation matrix; and a recursive calculation unit, configured to re-determine the second motion trajectory based on the latest obtained intrinsic and encoded value data when it is confirmed that the intrinsic and / or extrinsic parameters do not converge.

[0151] Based on the above embodiments, the camera and wheeled odometer are mounted on the wheeled mobile robot. When the wheeled mobile robot moves in a two-dimensional plane according to a preset path, the camera and wheeled odometer respectively collect image data and coded value data. The path is a sine path or a figure-eight path.

[0152] Based on the above embodiments, the external parameter calibration device further includes: an axis angle trajectory acquisition module, configured to extract the corresponding first axis angle motion trajectory and second axis angle motion trajectory from the first motion trajectory and the second motion trajectory respectively; and a delay time determination module, configured to register the first axis angle motion trajectory and the second axis angle motion trajectory, and determine the acquisition delay time between the camera and the wheel odometer based on the timestamp of the paired trajectory points.

[0153] Based on the above embodiments, the chassis increment determination module includes: a timestamp determination module, configured to determine the timestamp of the chassis trajectory point corresponding to the adjacent key frame based on the timestamp of the adjacent key frame and the delay time; and a chassis attitude acquisition module, configured to acquire the corresponding chassis trajectory point from the second motion trajectory based on the timestamp of the chassis trajectory point.

[0154] Based on the above embodiments, the first processing unit includes: a first pixel determination subunit, configured to obtain the pixel point at the relative position of the corresponding video frame according to the position information of the information box; a first pixel map acquisition subunit, configured to obtain a preset Gaussian weight matrix and obtain a pixel map composed of the pixel point and eight surrounding pixels; and a first Gaussian blur calculation subunit, configured to multiply the weights of the Gaussian weight matrix with the corresponding pixel values ​​in the pixel map and sum all the products to obtain the pixel value after Gaussian blur processing.

[0155] Based on the above embodiments, the linear solution module further includes: a first loop calculation unit, configured to calculate new intrinsic and extrinsic parameters through a first loop until the intrinsic and extrinsic parameters obtained in the first loop converge. The first loop includes updating the second motion trajectory based on the latest obtained intrinsic parameters, updating the second rotation increment and the second translation increment based on the new second motion trajectory, and updating the intrinsic and extrinsic parameters based on the new second rotation increment and the second translation increment; a second loop calculation unit, configured to calculate new intrinsic and extrinsic parameters through a second loop when the intrinsic and extrinsic parameters obtained in the first loop converge, until the intrinsic and extrinsic parameters obtained in the second loop converge. The second loop includes updating the second motion trajectory based on the latest obtained intrinsic parameters, updating the delay time based on the new second motion trajectory, and updating the intrinsic and extrinsic parameters through the first loop; and an optimal delay time calculation unit, configured to calculate the delay time using the latest obtained intrinsic parameters when the intrinsic and extrinsic parameters obtained in the second loop converge.

[0156] Based on the above embodiments, the extrinsic parameter calibration device further includes: a third motion trajectory determination module, configured to acquire motion data collected by the inertial measurement unit, fit the motion data and image data to form a third motion trajectory, wherein the camera trajectory points of the third motion trajectory include the velocity in the camera coordinate system and the acquisition timestamp; a chassis coordinate system velocity determination module, configured to determine the velocity of the wheel odometer corresponding to the previous acquisition timestamp based on the increment of the encoded value adjacent to the acquisition timestamp and the intrinsic parameter variable; a camera coordinate system velocity determination module, configured to determine the corresponding camera trajectory point in the third motion trajectory based on the acquisition timestamp of the encoded value, and correspondingly acquire the velocity in the camera coordinate system; a first error equation construction module, configured to transform the velocity in the camera coordinate system into the chassis coordinate system through the adjoint matrix of the extrinsic parameter variable, and construct a first error equation with the velocity in the chassis coordinate system corresponding to the acquisition timestamp; and a first optimization module, configured to use the newly obtained intrinsic and extrinsic parameters as the initial values ​​of the intrinsic and extrinsic parameter variables in the first error equation, and iteratively calculate new intrinsic and extrinsic parameters through the Levenberg-Marquadt rule until the intrinsic and extrinsic parameters converge.

[0157] Based on the above embodiments, the external parameter calibration device further includes: a third motion trajectory determination module, configured to acquire motion data collected by the inertial measurement unit, fit the motion data and image data to form a third motion trajectory, wherein the camera trajectory points of the third motion trajectory include the velocity, acceleration, and acquisition timestamp of the camera coordinate system; a chassis coordinate system velocity determination module, configured to determine the velocity of the chassis coordinate system of the wheel odometer corresponding to the previous acquisition timestamp based on the increment of the encoded value adjacent to the acquisition timestamp and the intrinsic parameter variable; a camera coordinate system velocity determination module, configured to determine the corresponding camera trajectory point in the third motion trajectory based on the acquisition timestamp of the encoded value, and correspondingly acquire the velocity and acceleration of the camera coordinate system; and a delay compensation velocity. The determination module is configured to multiply the acceleration of the camera coordinate system by the increment of the delay time variable, and add the velocity of the camera coordinate system to obtain the delay compensation velocity of the camera coordinate system. The second error equation construction module is configured to transform the delay compensation velocity of the camera coordinate system into the chassis coordinate system using the adjoint matrix of the extrinsic parameter variables, and construct the second error equation with the velocity of the chassis coordinate system corresponding to the acquisition timestamp. The second optimization module is configured to use the latest obtained intrinsic parameters, extrinsic parameters, and delay time as the initial values ​​of the intrinsic parameters, extrinsic parameters, and delay time variables in the second error equation, and iteratively calculate the intrinsic and extrinsic parameters using the Levenberg-Marquadt rule until the intrinsic parameters, extrinsic parameters, and delay time converge.

[0158] It is worth noting that in the above embodiments based on the external parameter calibration device, the various units and modules included are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be achieved; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the scope of protection of this application.

[0159] The external parameter calibration device provided in this application embodiment is included in the external parameter calibration equipment and can be used to execute the external parameter calibration method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0160] Figure 9 This is a schematic diagram of the structure of an external parameter calibration device provided in one embodiment of this application. Figure 9 As shown, the external parameter calibration device includes a processor 70, a memory 71, an input device 72, and an output device 73; the number of processors 70 in the external parameter calibration device can be one or more. Figure 9 Taking a processor 70 as an example; the processor 70, memory 71, input device 72, and output device 73 in the external parameter calibration device can be connected via a bus or other means. Figure 9 Taking the example of a connection between China and Israel via a bus.

[0161] The memory 71, as a computer-readable storage medium, can be used to store software programs, computer-executable programs, and modules, such as program instructions / modules corresponding to the extrinsic parameter calibration method in the embodiments of this application (e.g., the motion trajectory determination module 601, camera increment determination module 602, chassis increment determination module 603, and linear solution module 604 in the extrinsic parameter calibration device). The input device 72 can be used to receive input digital or character information, and generate key signal inputs related to user settings and function control of the extrinsic parameter calibration device. Specifically, the input device 72 receives image data acquired by the camera, coded value data acquired by the wheel odometer, and IMU data acquired by the IMU, and sends these data to the processor 70.

[0162] The processor 70 processes image data, encoded value data, and / or IMU data by running software programs, instructions, and modules stored in the memory 71, thereby implementing the aforementioned extrinsic parameter calibration method. The processor sends the extrinsic parameter calibration process and results to the output device 73, allowing the user to view the extrinsic parameter calibration progress through the output device.

[0163] The memory 71 may primarily include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function; the data storage area may store data created based on the use of the external parameter calibration device. Furthermore, the memory 71 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 71 may further include memory remotely located relative to the processor 70, which can be connected to the external parameter calibration device via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0164] The output device 73 may include audio output devices such as speakers and image output devices such as displays. Specifically, the display receives the external parameter calibration process and results sent by the processor 70, and displays the external parameter calibration processing progress on the screen.

[0165] The aforementioned external parameter calibration equipment includes an external parameter calibration device, which can be used to perform any external parameter calibration method and has corresponding functions and beneficial effects.

[0166] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, is used to perform the extrinsic parameter calibration method provided in the above embodiments.

[0167] Of course, the computer-readable storage medium provided in the embodiments of this application has computer-executable instructions that are not limited to the method operations described above, and can also perform related operations in the external parameter calibration method provided in any embodiment of this application.

[0168] Based on the above description of the implementation methods, those skilled in the art can clearly understand that this application can be implemented using software and necessary general-purpose hardware, and of course, it can also be implemented using hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory, hard disk, or optical disk, etc., including several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods of the various embodiments of this application.

[0169] Note that the above are merely preferred embodiments and the technical principles employed in this application. Those skilled in the art will understand that this application is not limited to the specific embodiments described herein, and various obvious changes, readjustments, and substitutions can be made without departing from the scope of protection of this application. Therefore, although this application has been described in detail through the above embodiments, this application is not limited to the above embodiments, and may include many other equivalent embodiments without departing from the concept of this application, the scope of which is determined by the scope of the appended claims.

Claims

1. A method for calibrating an extrinsic parameter, characterized in that, The method comprises the following steps: acquiring image data collected by a camera and an encoding value collected by a wheel odometer, determining a first motion trajectory of the camera according to the image data, and determining a second motion trajectory of the wheel odometer according to the encoding value and a preset wheel odometer internal parameter; acquiring all adjacent key frames in the first motion trajectory, calculating a first rotation increment and a first translation increment between the adjacent key frames, and determining that the first rotation increment or the first translation increment between the adjacent key frames is greater than a corresponding preset increment threshold; determining corresponding chassis trajectory points in the second motion trajectory according to the acquisition time stamps of the adjacent key frames, and calculating a second rotation increment and a second translation increment between the corresponding chassis trajectory points; based on a preset hand-eye calibration equation set, recursively calculating a new internal parameter and an external parameter according to the newly obtained first rotation increment, first translation increment, second rotation increment and second translation increment until the internal parameter and the external parameter converge.

2. The method of claim 1, wherein, The method based on the preset hand-eye calibration equation set, recursively calculating a new internal parameter and an external parameter according to the newly obtained first rotation increment, first translation increment, second rotation increment and second translation increment until the internal parameter and the external parameter converge, comprises: determining a rotation axis vector of the camera according to the first rotation increment, aligning the rotation axis vector with a rotation axis vector of the preset wheel odometer, and determining a pitch angle and a roll angle of an external parameter rotation matrix; determining a new second motion trajectory according to the newly obtained internal parameter and the encoding value, acquiring corresponding chassis trajectory points from the newly obtained second motion trajectory according to the acquisition time stamps of the adjacent key frames, and calculating a new second rotation increment and a second translation increment according to the chassis trajectory points; substituting the newly obtained first rotation increment, first translation increment, second rotation increment and second translation increment into the hand-eye calibration equation set to obtain a new internal parameter and a yaw angle of a new external parameter rotation matrix and an external parameter translation matrix; when it is confirmed that the internal parameter and / or the external parameter do not converge, re-determining a second motion trajectory according to the newly obtained internal parameter and the encoding value.

3. The method of claim 1, wherein, The camera and the wheel odometer are installed on a wheeled mobile robot, and the camera and the wheel odometer respectively collect the image data and the encoding value when the wheeled mobile robot moves according to a preset path in a two-dimensional plane, and the path is a sine path or an eight-character path.

4. The method of claim 1, wherein, Before the step of determining the corresponding chassis trajectory points in the second motion trajectory according to the acquisition time stamps of the adjacent key frames, the method further comprises the following steps: extracting a corresponding first axis-angle motion trajectory and a second axis-angle motion trajectory from the first motion trajectory and the second motion trajectory respectively; aligning the first axis-angle motion trajectory and the second axis-angle motion trajectory, and determining a collection delay time between the camera and the wheel odometer according to the time stamps of the paired trajectory points.

5. The method of claim 4, wherein, The step of determining the corresponding chassis trajectory points in the second motion trajectory according to the acquisition time stamps of the adjacent key frames comprises the following steps: determining the acquisition time stamps of the chassis trajectory points corresponding to the adjacent key frames according to the acquisition time stamps of the adjacent key frames and the delay time; According to a collection time stamp of the chassis track point, a corresponding chassis track point is obtained from the second motion track.

6. The method of claim 5, wherein, The new intrinsic parameter and the extrinsic parameter are recursively calculated according to the newly obtained first rotation increment, the first translation increment, the second rotation increment and the second translation increment based on the preset hand-eye calibration equation set until the intrinsic parameter and the extrinsic parameter converge, including: The new intrinsic parameter and the extrinsic parameter are calculated through a first loop until the intrinsic parameter and the extrinsic parameter obtained by the first loop converge, and the first loop includes updating the second motion track according to the newly obtained intrinsic parameter, updating the second rotation increment and the second translation increment according to the new second motion track, and updating the intrinsic parameter and the extrinsic parameter according to the new second rotation increment and the second translation increment; When the intrinsic parameter and the extrinsic parameter obtained by the first loop converge, the new intrinsic parameter and the extrinsic parameter are calculated through a second loop until the intrinsic parameter and the extrinsic parameter obtained by the second loop converge, and the second loop includes updating the second motion track according to the newly obtained intrinsic parameter, updating the delay time according to the new second motion track, and updating the intrinsic parameter and the extrinsic parameter through the first loop; When the intrinsic parameter and the extrinsic parameter obtained by the second loop converge, the delay time is calculated through the newly obtained intrinsic parameter.

7. The method of claim 1, wherein, After the intrinsic parameter and the extrinsic parameter converge, further including: Obtaining movement data collected by an inertial measurement unit, fitting the movement data and the image data to obtain a third motion track, and the camera track point of the third motion track including the speed of the camera coordinate system and the collection time stamp; According to the adjacent encoding value increment and the intrinsic parameter variable of the collection time stamp, the speed of the chassis coordinate system of the wheel odometer corresponding to the previous collection time stamp is determined; According to the collection time stamp of the encoding value, the corresponding camera track point in the third motion track is determined, and the speed of the camera coordinate system is obtained; Through the adjoint matrix of the extrinsic parameter variable, the speed of the camera coordinate system is converted to the chassis coordinate system, and a first error equation is constructed with the speed of the chassis coordinate system corresponding to the collection time stamp; The newly obtained intrinsic parameter and extrinsic parameter are taken as the initial values of the intrinsic parameter variable and the extrinsic parameter variable in the first error equation, and the new intrinsic parameter and extrinsic parameter are iteratively calculated through the Levenberg-Marquadt rule until the intrinsic parameter and the extrinsic parameter converge.

8. The method of claim 6, wherein, After the delay time is calculated through the newly obtained intrinsic parameter when the intrinsic parameter and the extrinsic parameter obtained by the second loop converge, further including: Obtaining movement data collected by an inertial measurement unit, fitting the movement data and the image data to obtain a third motion track, and the camera track point of the third motion track including the speed, acceleration and collection time stamp of the camera coordinate system; According to the adjacent encoding value increment and the intrinsic parameter variable of the collection time stamp, the speed of the chassis coordinate system of the wheel odometer corresponding to the previous collection time stamp is determined; According to the collection time stamp of the encoding value, the corresponding camera track point in the third motion track is determined, and the speed and acceleration of the camera coordinate system are obtained; The acceleration of the camera coordinate system is multiplied by the increment of the delay time variable, and then the speed of the camera coordinate system is added to obtain the delay compensation speed of the camera coordinate system; The delay compensation speed of the camera coordinate system is converted to the chassis coordinate by an adjoint matrix of the external parameter variable, and a second error equation is constructed with the speed of the chassis coordinate corresponding to the acquisition timestamp; The latest obtained intrinsic parameter, external parameter and delay time are taken as initial values of the intrinsic parameter variable, external parameter variable and delay time variable in the second error equation, and the intrinsic parameter and external parameter are iteratively calculated by Levenberg-Marquadt rule until the intrinsic parameter, the external parameter and the delay time converge.

9. An external parameter calibration device, characterized in that Comprise: A motion trajectory determination module configured to obtain image data collected by a camera and an encoding value collected by a wheeled odometer, determine a first motion trajectory of the camera according to the image data, and determine a second motion trajectory of the wheeled odometer according to the encoding value and a preset intrinsic parameter of the wheeled odometer; A camera increment determination module configured to obtain all adjacent key frames in the first motion trajectory, calculate a first rotation increment and a first translation increment between the adjacent key frames, and the first rotation increment or the first translation increment between the adjacent key frames is greater than a corresponding preset increment threshold; A chassis increment determination module configured to determine corresponding chassis trajectory points in the second motion trajectory according to acquisition timestamps of the adjacent key frames, and calculate a second rotation increment and a second translation increment between the corresponding chassis trajectory points; A linear solution module configured to recursively calculate new intrinsic parameters and external parameters according to the latest obtained first rotation increment, first translation increment, second rotation increment and second translation increment based on a preset hand-eye calibration equation set, until the intrinsic parameters and the external parameters converge.

10. An extrinsic parameter calibration device, characterized by Comprise: A processor, a memory, an input device and an output device, the processor, the memory, the input device and the output device are connected, wherein: The memory is used for storing software programs, instructions and modules; The input device is used for receiving input digital or character information and sending the digital or character information to the processor; The processor is used for running the software programs, instructions and modules stored in the memory, realizing the external parameter calibration method according to the digital or character information, and sending the processing result to the output device; The output device is used for outputting images or sounds according to the processing result.

11. A computer readable storage medium, characterized in that, A computer program is stored thereon, which is executed by a processor to realize the external parameter calibration method according to any one of claims 1-8. A computer program is stored thereon, which is executed by a processor to realize the external parameter calibration method according to any one of claims 1-8.

Images