Calibration method and system of visual radar, visual radar and medium

Abstract

The application discloses a kind of visual radar's calibration method, calibration system, visual radar and medium, it is related to calibration technical field, in pre-set closed-loop control state, the relative pose parameters under each pre-set angle position are acquired, the relative pose parameters are input into visual radar calibration program, the shaft direction of visual radar that can be applied to equipment can be quickly and accurately output, the position and posture of encoder in visual radar relative to inertial measurement unit, and the position of camera in visual radar relative to encoder, solve the problem that visual radar device cannot determine the shaft position and direction of each angle position.

Description

Technical Field

[0001] This invention relates to the field of calibration technology, and in particular to a calibration method, calibration system, visual radar, and medium for a visual radar. Background Technology

[0002] Monocular cameras are commonly mounted on various automated devices such as autonomous vehicles, drones, and ground-based autonomous navigation robots. These monocular cameras are typically used in conjunction with onboard IMUs to form monocular vision radar devices, applying monocular SLAM algorithms for real-time localization and mapping. However, existing monocular cameras are often simply mounted directly on various devices, and the camera's field of view is limited by factors such as the angle of view, mechanical obstructions, and installation location, sometimes preventing the SLAM algorithm from achieving its full performance potential.

[0003] While some solutions exist to address the aforementioned issues, such as mounting a visual inertial system on the gimbal, these solutions introduce new problems for SLAM algorithms on various devices. The main issue is that some gimbals cannot achieve cyclic rotation, and existing gimbal calibration methods struggle to determine the position and direction of the rotation axis. This makes it difficult to transfer the pose obtained after running the SLAM algorithm on a gimbal equipped with a visual inertial system to the mounted device.

[0004] Application content

[0005] The main objective of this invention is to provide a calibration method, calibration system, visual radar, and medium for visual radar, aiming to solve the technical problem that visual radar devices have difficulty in determining their position and orientation.

[0006] To achieve the above objectives, the present invention provides a calibration method for a visual radar, the calibration method comprising the following steps:

[0007] Under the preset closed-loop control state, the relative pose parameters of the camera and the inertial measurement unit at various preset angle positions of the visual radar are acquired;

[0008] The relative pose parameters and the preset angle positions corresponding to each relative pose parameter are input into the visual radar calibration program, and the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder are output.

[0009] Optionally, before the step of acquiring the relative pose parameters of the camera and inertial measurement unit at various preset angular positions under the preset closed-loop control state, the calibration method includes:

[0010] Obtain the motor rotor position;

[0011] The preset closed-loop control state is set based on the motor rotor position.

[0012] Optionally, the step of acquiring the relative pose of the camera and the inertial measurement unit at various preset angular positions of the visual radar includes:

[0013] Obtain camera calibration results and inertial measurement unit calibration results;

[0014] Based on the camera calibration results and the inertial measurement unit calibration results, the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position are obtained.

[0015] Optionally, before the step of inputting the preset angle positions corresponding to the relative pose parameters and each of the relative position parameters into the visual radar calibration program, the calibration method further includes:

[0016] Obtain a set of variables to be optimized, consisting of preset angular positions, the rotation direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder; and obtain state estimates of the set of variables to be optimized through a preset state estimation function.

[0017] Optionally, the step of inputting the preset angle positions corresponding to the relative pose parameters and each of the relative position parameters into the visual radar calibration program, and outputting the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder, includes:

[0018] Substitute the relative pose parameters and the preset angle positions corresponding to each relative pose parameter into the visual radar calibration program to obtain the parameter input set;

[0019] The parameter input set, the set of variables to be optimized, and the state estimate are respectively substituted into the visual radar calibration program to construct the attitude residual model and the position residual model.

[0020] The partial derivatives of the set of variables to be optimized are calculated based on the attitude residual model and the position residual model.

[0021] The partial derivatives are imported into a preset optimization framework to solve for the rotation axis direction, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder.

[0022] The present invention also provides a calibration system for a visual radar, the calibration system comprising:

[0023] The sensor calibration module is used to acquire the relative pose parameters of the camera and the inertial measurement unit at various preset angle positions of the visual radar under preset closed-loop control conditions.

[0024] The visual radar calibration module is used to input the relative pose parameters and the preset angle positions corresponding to each of the relative pose parameters into the visual radar calibration program, and output the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder.

[0025] Optionally, the sensor calibration module includes a camera calibration module, an inertial measurement unit calibration module, and a camera-inertial measurement unit joint calibration module that is connected to the output terminals of the camera calibration module and the inertial measurement unit calibration module, respectively.

[0026] The camera calibration module is used to calibrate the calibration photos transmitted from the visual radar device and generate camera calibration results;

[0027] The inertial measurement unit calibration module is used to calculate the inertial measurement data transmitted from the visual radar device to obtain the inertial measurement unit calibration result.

[0028] The camera-inertial measurement unit joint calibration module is used to obtain the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position based on the camera calibration results and the inertial measurement unit calibration results.

[0029] Optionally, the calibration system of the visual radar establishes a communication connection with the visual radar device. The visual radar device includes a global shutter camera, a micro slip ring, a micro hollow DC brushless motor, and a base connected in sequence. The micro slip ring is also connected to the base.

[0030] The base includes a microcontroller control board, a magnetic field orientation control drive board that is bidirectionally connected to the microcontroller control board, and a processing unit that is bidirectionally connected to both the microcontroller control board and the magnetic field orientation control drive board. The processing unit is also connected to the micro slip ring.

[0031] The microcontroller control board is equipped with an inertial measurement unit, and the magnetic field orientation control drive board is equipped with an encoder.

[0032] The present invention also provides a visual radar, the visual radar comprising: a memory, a processor, and a computer processing program stored in the memory and executable on the processor, wherein the computer processing program, when executed by the processor, implements the steps of the above-described visual radar calibration method.

[0033] The present invention also provides a computer-readable storage medium storing a computer processing program, which, when executed by a processor, implements the steps of the above-described visual radar calibration method.

[0034] The visual radar calibration method proposed in this invention obtains the relative pose parameters of the camera and inertial measurement unit at various preset angular positions of the visual radar under a preset closed-loop control state. By maintaining the electrode angle of the visual radar at a certain preset angular position under this state, the continuous change in angular position during the relative pose parameter calculation process avoids the cumulative error caused by such changes. The relative pose parameters and their corresponding preset angular positions are input into the visual radar calibration program, which outputs the rotation axis direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder. Since the relative pose parameters are calculated at each fixed preset angular position, the obtained relative pose parameters, through a programmed visual radar calibration program, can quickly and accurately output the rotation axis direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder, thus solving the problem that the visual radar device cannot determine the rotation axis position and direction at various angular positions. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention;

[0036] Figure 2 This is a flowchart illustrating the first embodiment of the calibration method for the visual radar of the present invention;

[0037] Figure 3 This is a flowchart illustrating the second embodiment of the calibration method for the visual radar of the present invention;

[0038] Figure 4 This is a schematic diagram of the calibration system for the visual radar of the present invention;

[0039] Figure 5 This is a schematic diagram of the structure of the visual radar device of the present invention.

[0040] The realization of the objective, functional features and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0041] It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0042] like Figure 1 As shown, Figure 1 This is a schematic diagram of the terminal structure of the hardware operating environment involved in the embodiments of the present invention.

[0043] The calibration method for visual radar in this embodiment of the invention is applied to a visual radar, such as... Figure 1 As shown, the visual radar may include: a processor 1001, such as a CPU; a network interface 1004; a user interface 1003; a memory 1005; and a communication bus 1002. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display area and an input unit such as a keyboard. Optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be high-speed RAM or non-volatile memory, such as a disk drive. Optionally, the memory 1005 may also be a storage device independent of the aforementioned processor 1001.

[0044] Optionally, visual radar may also include a camera, RF (Radio Frequency) circuitry, sensors, audio circuitry, a WiFi module, and so on. These sensors may include light sensors, motion sensors, and other sensors. Specifically, light sensors may include ambient light sensors and proximity sensors. The ambient light sensor can adjust the display brightness according to the ambient light level, while the proximity sensor can turn off the display and / or backlight when the mobile terminal is moved to the ear. As a type of motion sensor, a gravity accelerometer can detect the magnitude of acceleration in various directions (generally three axes). When stationary, it can detect the magnitude and direction of gravity, and can be used for applications that identify the mobile terminal's posture (such as landscape / portrait switching, related games, magnetometer posture calibration), vibration recognition functions (such as pedometers, taps), etc. Of course, mobile terminals may also be equipped with other sensors such as gyroscopes, barometers, hygrometers, thermometers, and infrared sensors, which will not be elaborated upon here.

[0045] Those skilled in the art will understand that Figure 1 The visual radar structure shown does not constitute a limitation on visual radar and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0046] like Figure 1 As shown, the memory 1005, which serves as a computer storage medium, may include an operating system, a network communication module, a user interface module, and a computer processing program.

[0047] exist Figure 1 In the terminal shown, network interface 1004 is mainly used to connect to the backend server and communicate data with it; user interface 1003 is mainly used to connect to the client (user terminal) and communicate data with it; while processor 1001 can be used to call the computer processing program stored in memory 1005 and perform the following operations:

[0048] Under the preset closed-loop control state, the relative pose parameters at each preset angle position are obtained;

[0049] The relative pose parameters and the preset angle positions corresponding to each relative pose parameter are input into the visual radar calibration program in the visual radar calibration device, and the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder are output.

[0050] Furthermore, the processor 1001 can call the computer processing program stored in the memory 1005 and also perform the following operations:

[0051] Before obtaining the relative pose parameters at each preset angle position under the preset closed-loop control state, the motor speed is calculated based on the input motor rotor electrical angle and angular velocity acceleration.

[0052] The preset closed-loop control state is set based on the motor speed.

[0053] Furthermore, the processor 1001 can call the computer processing program stored in the memory 1005 and also perform the following operations:

[0054] Before obtaining the relative pose parameters at each preset angle position, obtain the camera calibration results and the inertial measurement unit calibration results;

[0055] The camera calibration results and the inertial measurement unit calibration results are synchronized to obtain synchronization data, and the relative pose parameters of the visual radar at each preset angle position are obtained based on the synchronization data.

[0056] Furthermore, the processor 1001 can call the computer processing program stored in the memory 1005 and also perform the following operations:

[0057] Before the step of inputting the relative pose parameters and the preset angle positions corresponding to each of the relative position parameters into the visual radar calibration program, a set of variables to be optimized is obtained, consisting of each preset angle position, the rotation axis direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder; and the state estimate value of the set of variables to be optimized is obtained through a preset state estimation function.

[0058] Furthermore, the processor 1001 can call the computer processing program stored in the memory 1005 and also perform the following operations:

[0059] The steps of inputting the relative pose parameters and the preset angle positions corresponding to each of the relative position parameters into the visual radar calibration program, and outputting the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder, include: substituting the relative pose parameters and the preset angle positions corresponding to each of the relative pose parameters into the visual radar calibration program to obtain a parameter input set;

[0060] The parameter input set, the set of variables to be optimized, and the state estimate are respectively substituted into the visual radar calibration program to construct the attitude residual model and the position residual model.

[0061] The partial derivatives of the set of variables to be optimized are calculated based on the attitude residual model and the position residual model.

[0062] The partial derivatives are imported into a preset optimization framework to solve for the rotation axis direction, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder.

[0063] This invention provides a calibration method for a visual radar. In the first embodiment of the visual radar calibration method, referring to... Figure 2 The method includes:

[0064] Step S10: Under the preset closed-loop control state, acquire the relative pose parameters of the camera and inertial measurement unit at various preset angle positions of the visual radar.

[0065] Because conventional visual radar cannot achieve cyclic rotation, the pose parameters obtained based on conventional visual radar have accumulated errors, which in turn makes conventional visual radar devices unable to accurately determine the position and direction of the rotation axis at various angles.

[0066] To address the aforementioned issues, this embodiment proposes acquiring the relative pose parameters of the visual radar at various preset angles under a preset closed-loop control state. This preset closed-loop control state is a control state for the visual radar. In this state, the motor within the visual radar rotates cyclically according to a fixed preset angle position. This ensures that the visual radar can accurately acquire the relative pose parameters of the electrodes at a specific preset angle position, avoiding incorrect relative pose parameters obtained due to inconsistent rotation angles, which could affect subsequent operations.

[0067] Step S20: Input the relative pose parameters and the preset angle positions corresponding to each of the relative pose parameters into the visual radar calibration program, and output the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder.

[0068] It should be noted that the visual radar calibration program in this embodiment is pre-set in the visual radar calibration device and adopts a least squares-based calibration method. It can automatically calculate and output the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder based on the input relative pose parameters and the preset angle positions corresponding to each relative pose parameter.

[0069] After calculating the relative pose parameters of the visual radar at each preset angle position through step S10, all the obtained relative pose parameters and preset angle positions are input into the visual radar calibration program. At this time, the visual radar calibration program automatically outputs and applies the relative pose parameters and preset angle positions according to the model formula in its program, thereby outputting the final result, namely the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder. Because the input relative pose parameters are obtained at a fixed preset angle position, the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder calculated based on the relative pose parameters can be applied to the device, thereby enabling the device equipped with the visual radar to perform accurate positioning and navigation.

[0070] Optionally, before inputting the preset angle positions corresponding to the relative pose parameters and each of the relative position parameters into the visual radar calibration program in step S20, the calibration method further includes:

[0071] Step S201: Obtain the set of variables to be optimized, which consists of each preset angle position, the rotation direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder; and obtain the state estimate value of the set of variables to be optimized through a preset state estimation function.

[0072] First, the set of preset angle positions, the rotation direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position set of the camera relative to the encoder are obtained to obtain the set of variables to be optimized as shown in Formula 1.

[0073]

[0074] Among them, X c This represents the set of variables to be optimized for visual radar. Let θ1 be the variable to be optimized, and θ be the variable to be optimized. n Indicates the preset angle position of the visual radar, a T This refers to the rotating shaft of the visual radar. This indicates the attitude of the encoder coordinate system in the visual radar relative to the inertial measurement unit coordinate system in the visual radar. This indicates the position of the encoder coordinate system in the visual radar relative to the inertial measurement unit coordinate system in the visual radar. This indicates the position of the camera coordinate system in the visual radar relative to the encoder coordinate system in the visual radar.

[0075] Meanwhile, to make the optimization process of the later-established least squares optimization problem faster and more accurate, the state estimates of the rotation axis direction and the encoder's attitude relative to the inertial measurement unit in the set of variables to be optimized can be obtained through Equation 2:

[0076]

[0077] Among them, a ∧ This represents the state estimate of the axis of rotation. Since the preset angle positions in this example are 0 degrees, 90 degrees, and 270 degrees, the preset angle position substituted into Formula 5 is... and 0, I R C and I R E This represents the state estimate of the attitude, where the position estimate is set to 0.

[0078] Optionally, step S20, which involves inputting the relative pose parameters and the preset angle positions corresponding to each of the relative position parameters into the visual radar calibration program, and outputting the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder, includes:

[0079] Step S202: Substitute the relative pose parameters and the preset angle positions corresponding to each relative pose parameter into the visual radar calibration program to obtain a parameter input set.

[0080] The calculation process in the visual radar calibration program is as follows.

[0081] First, substitute each parameter in the relative pose parameters into the quantity to be optimized in the first problem to obtain a set of parameter inputs with actual parameter values.

[0082] Step S203: Substitute the parameter input set, the set of variables to be optimized, and the state estimate into the visual radar calibration program to construct the attitude residual model and the position residual model.

[0083] In the setup phase of the visual radar calibration procedure, a least-squares optimization problem for visual radar calibration is first constructed. Two problems are constructed here. The first problem describes the quantity to be optimized in the optimization problem, as shown in Formula 1:

[0084] The first problem describes the attitude residuals in the least squares optimization problem, as shown in Equation 3:

[0085] r 1i (z 1i ,X c )=(exp(θ i a^) I R E - I R C (θ))p————Formula 3

[0086] Where, r 1i (z i ,X c ) represents the attitude residual, z 1i The observed values ​​representing the attitude, (exp(θ) i a^) I R E The formula for calculating the observed values ​​representing attitude. I R C (θ) represents the formula for calculating the estimated attitude value, and p represents the projection vector.

[0087] The second problem describes the location residual in the least squares optimization problem, as shown in Equation 4:

[0088] r 2i (z 2i ,X c )= I p E +exp(θ i a^) E p C - I p C (θ)————Formula 4

[0089] Where, r 2i (z 2i ,X c ) represents the positional residual, z 2i Observations representing location, I p E The projection vector of the encoder coordinate system relative to the inertial measurement unit coordinate system is exp(θ). i a^) E p C The formula for calculating the observed value of the location is shown below. I p C (θ) represents the formula for calculating the estimated position.

[0090] Integrating Equations 3 and 4, we obtain the least squares optimization problem as shown in Equation 5:

[0091]

[0092] in, Let r represent the least squares optimization problem. 1i (z 1i ,X c ) represents the attitude residual, P1 and P2 represent the weight matrices, and r 2i (z 2i ,X c ) represents the positional residual.

[0093] After obtaining the parameter input set, rotation axis direction, position and attitude, and state estimates through steps S201 and S203, the parameter input set, rotation axis direction, position and attitude, and state estimates are substituted into the corresponding positions in formulas 3 and 4, respectively, to obtain the attitude residual model (i.e., formula 6) and the position residual model (i.e., formula 7):

[0094] e1=||r1||2=||ΔR p ||2————Formula 6

[0095] e2=||r2||2=||ΔRP ||2————Formula 7

[0096] Step S204: Calculate the partial derivatives of the set of variables to be optimized based on the attitude residual model and the position residual model.

[0097] Specifically, the partial derivatives of the camera's position relative to the encoder, the rotation direction of the visual radar, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder are calculated for each preset angle position.

[0098] In this embodiment, the partial derivatives of attitude are solved using a right perturbation model on the SO3 manifold, and the partial derivatives of position are solved using a perturbation model on the spherical manifold. Specifically, the right perturbation model on the SO3 manifold is shown in Equation 8:

[0099]

[0100] in, and I represents the partial derivative of the attitude residual with respect to the variable to be optimized, where I is the third unit.

[0101] The perturbation model on the spherical manifold is shown in Equation 9:

[0102]

[0103] in, and This represents the partial derivative of the location residual with respect to the variable to be optimized.

[0104] Step S205: The partial derivatives are imported into a preset optimization framework to solve for the rotation axis direction, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder.

[0105] It should be noted that the preset optimization framework in this embodiment is a least-squares optimization framework built using the Ceres library. The attitude partial derivative and position partial derivative obtained in step S204 are solved to obtain the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder.

[0106] In this embodiment, under a preset closed-loop control state, the relative pose parameters of the camera and inertial measurement unit at various preset angular positions of the visual radar are acquired. By using the visual radar under the preset closed-loop control state, the electrode angle of the visual radar can be kept constant at a certain preset angular position, thus avoiding the cumulative error in the calculated relative pose parameters caused by continuous changes in angular position during the relative pose parameter calculation process. The relative pose parameters are input into the visual radar calibration program, which outputs the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder. Since the relative pose parameters are calculated at each fixed preset angular position, the acquired relative pose parameters, through a programmed visual radar calibration program, can quickly and accurately output the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder, which can be applied to the device. This solves the problem that the visual radar device cannot determine the rotation axis position and direction at various angular positions.

[0107] Furthermore, based on the first embodiment of the calibration method for the visual radar of the present invention described above, a second embodiment of the calibration method for the visual radar of the present invention is proposed.

[0108] Reference Figure 3 In the second embodiment of the calibration method for the visual radar of the present invention, before the step S10 above, in which the relative pose parameters of the camera and the inertial measurement unit of the visual radar are obtained at various preset angular positions under a preset closed-loop control state, the calibration method includes:

[0109] Step S101: Obtain the motor rotor position.

[0110] Step S102: Set the preset closed-loop control state based on the motor rotor position.

[0111] In this embodiment, the preset closed-loop control state needs to be set based on the visual radar. Specifically, the encoder in the visual radar reads the rotor position of the miniature hollow brushless DC motor in the visual radar.

[0112] At this time, the encoder sends the motor rotor position as a control reference value to the field orientation control drive board. The field orientation control drive board controls the angular position of the motor rotation according to the motor rotor position, so as to realize the operation of the preset closed-loop control state.

[0113] Optionally, the step of obtaining the relative pose of the camera and the inertial measurement unit at various preset angular positions in step S10 includes:

[0114] Step S103: Obtain the camera calibration results and the inertial measurement unit calibration results.

[0115] Step S104: Based on the camera calibration results and the inertial measurement unit calibration results, obtain the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position.

[0116] The acquisition of relative pose parameters is divided into three steps: camera calibration, inertial measurement unit calibration, and camera-inertial measurement unit joint calibration.

[0117] Specifically, the camera calibration steps are as follows: In this embodiment, a black and white checkerboard calibration board is used, and the Zhang Zhengyou calibration method is used for calibration. After printing the black and white checkerboard into a rigid foam board and measuring the length of each individual square, several photos of the black and white checkerboard calibration board are taken from various angles based on the global shutter camera in the visual radar. The photos are then imported into the camera calibration module that calibrates the photos, and the camera calibration results of the photos are obtained through the camera calibration module.

[0118] The inertial measurement unit (IMU) calibration steps are as follows: After the visual radar is left to stand still for a preset number of hours (approximately 3 hours in this embodiment), the inertial measurement data of the sensor in the visual radar is acquired. This inertial measurement data is then transmitted to the IMU calibration module, which calibrates the inertial measurement data. The IMU calibration result is then obtained through the IMU calibration module.

[0119] The camera-inertial measurement unit (IMU) joint calibration steps are as follows: With the visual radar's motor angle at a fixed preset angle, move the visual radar back and forth three times in each direction and angle—up / down, left / right, forward / backward, pitch, roll, and yaw—in front of the Apriltag calibration board. Simultaneously, the ARM microcomputer in the visual radar collects and records the ROS packages of the IMU, encoder, and camera. Import the collected ROS packages into the Kalibr toolbox within the camera-inertial measurement unit joint calibration module. The Kalibr toolbox synchronously runs based on the imported camera and IMU calibration results to obtain synchronization data. Based on the synchronization data and the ROS packages, calculations are performed to obtain the relative pose parameters at a fixed preset angle position. The preset angle position can then be changed, and the above steps repeated to obtain the relative pose parameters at the changed preset angle position. This must include at least three preset angle positions: 0°, 90°, and 270°.

[0120] In this embodiment, by obtaining the camera calibration results and the inertial measurement unit calibration results, the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position are obtained, thereby improving the accuracy and reliability of the obtained relative pose parameters.

[0121] As shown in the figure Figure 4 This is a schematic diagram of the calibration system for the visual radar of the present invention. The calibration system for the visual radar includes:

[0122] The sensor calibration module 10 is used to acquire the relative pose parameters of the camera and the inertial measurement unit at various preset angle positions of the visual radar under a preset closed-loop control state.

[0123] The visual radar calibration module 20 is used to input the relative pose parameters into the visual radar calibration program, and output the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder.

[0124] The visual radar calibration module 20 uses a visual radar calibration toolkit for calibration. It takes the relative pose parameters at each preset angle position collected in the sensor calibration module 10 as input and calculates the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera in the visual radar relative to the encoder in a least squares optimization manner.

[0125] Furthermore, the sensor calibration module 10 includes a camera calibration module, an inertial measurement unit calibration module, and a camera-inertial measurement unit joint calibration module that is connected to the output terminals of the camera calibration module and the inertial measurement unit calibration module, respectively.

[0126] The camera calibration module 30 is used to calibrate the calibration photos transmitted from the visual radar device and generate camera calibration results.

[0127] The camera calibration module 30 uses a black and white checkerboard calibration plate and Zhang Zhengyou calibration method for calibration. The specific calculation and processing steps of the camera calibration results are shown in step S103, and will not be repeated here.

[0128] The inertial measurement unit calibration module 40 is used to calculate the inertial measurement data transmitted from the visual radar device to obtain the inertial measurement unit calibration result.

[0129] The inertial measurement unit calibration module 40 uses the allan_variance_ros toolbox. The specific calculation and processing steps for the inertial measurement unit calibration results are shown in step S103, and will not be repeated here.

[0130] The camera-inertial measurement unit joint calibration module 50 is used to obtain the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position based on the camera calibration results and the inertial measurement unit calibration results.

[0131] The camera-inertial measurement unit joint calibration module 50 uses the Apriltag calibration board and is calibrated using the Kalibr toolbox. The specific calculation and processing steps for the relative pose parameters are shown in step S104, and will not be repeated here.

[0132] Specifically, refer to Figure 5 As shown, the calibration system of the visual radar establishes a communication connection with the visual radar device. The visual radar device includes a global shutter camera 1, a miniature slip ring 2, a miniature hollow DC brushless motor 3, and a base 4 connected in sequence. The miniature slip ring 2 is also connected to the base 4.

[0133] The base 4 includes a microcontroller control board 5, a magnetic field orientation control drive board 6 that is bidirectionally connected to the microcontroller control board 5, and a processing unit 7 that is bidirectionally connected to both the microcontroller control board 5 and the magnetic field orientation control drive board 6. The processing unit 7 is also connected to the micro slip ring 2.

[0134] Furthermore, the microcontroller control board 5 is equipped with an inertial measurement unit (IMU), and the magnetic field orientation control drive board 6 is equipped with an encoder.

[0135] In this embodiment, the processing unit 7 is an ARM microcomputer, and the visual radar device will be described here.

[0136] Combination Figure 5 As shown, the global shutter camera 1, miniature slip ring 2, miniature hollow brushless DC motor 3, and base 4 are mechanically connected together. The microcontroller control board 5, the magnetic field orientation control drive board 6, and the processing unit 7 are connected via an RS485 bus for serial bus communication. The magnetic field orientation control drive board 6 is connected to the miniature hollow brushless DC motor 3, outputting three-phase current to achieve magnetic field orientation control of the motor. The visual signal acquired by the global shutter camera 1 is connected to the processing unit 7 via the miniature slip ring 5 and transmitted to the processing unit 7 via USB 2.0 communication.

[0137] The inertial measurement unit (IMU) outputs angular velocity and acceleration signals, which are transmitted to the RS485 bus via the microcontroller control board 5 for processing unit 7 to read. Alternatively, it can directly output angular velocity and acceleration signals to the microcontroller control board 5 for closed-loop control. The encoder can output rotation angle signals, which are used to achieve field-oriented control on the field-oriented control drive board 6. Closed-loop control can also be performed on the field-oriented control drive board 6, which transmits the rotation angle signals output by the encoder to the RS485 bus for reading by the microcontroller control board 5 and processing unit 7.

[0138] It should be noted that the visual radar also includes a power management module 8, which is used to supply power to the global shutter camera 1, the miniature hollow DC brushless motor 3, the microcontroller control board 5, the magnetic field orientation control drive board 6, the processing unit 7, etc.

[0139] Compared to conventional visual radar devices composed of a monocular camera and an inertial measurement unit (IMU), the visual radar device in this embodiment ensures the continuity of motor rotation based on the mounted miniature slip ring 2 and miniature hollow DC brushless motor 3. At the same time, it adopts a structure in which the IMUs are separated from each other. The relative attitude between the IMU and the camera can be estimated by the electric angle of the motor rotor. This ensures that while the motor drives the global shutter camera 1 to rotate continuously, the IMU does not rotate, thereby improving the dynamic response accuracy of the IMU and reducing the cumulative error.

[0140] Furthermore, this embodiment of the invention also proposes a visual radar, which includes a memory, a processor, and a computer processing program stored in the memory and executable on the processor. When the processor executes the computer processing program, it implements the steps of the above-mentioned visual radar calibration method.

[0141] Furthermore, the present invention also proposes a computer-readable storage medium storing a computer processing program, wherein the modeling and simulation program, when executed by a processor, implements the steps of the above-described visual radar calibration method.

[0142] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0143] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0144] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to conventional technology, can be embodied in the form of a software product. This computer software product is stored in a computer-readable storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of the present invention.

[0145] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural or procedural changes made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A calibration method for visual radar, characterized in that, The calibration method for visual radar is applied to a calibration system for visual radar, and the calibration method for visual radar includes the following steps: Under a preset closed-loop control state, the relative pose parameters of the camera and the inertial measurement unit of the visual radar at various preset angle positions are acquired. In the preset closed-loop control state, the motor in the visual radar rotates cyclically according to a fixed preset angle position to ensure that the visual radar can accurately acquire the relative pose parameters of the motor at a certain preset angle position. Obtain a set of variables to be optimized, consisting of each preset angle position, the rotation direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera relative to the encoder. The state estimate of the set of variables to be optimized is obtained by using a preset state estimation function; Substitute the relative pose parameters and the preset angle positions corresponding to each relative pose parameter into the visual radar calibration program to obtain the parameter input set; The parameter input set, the set of variables to be optimized, and the state estimate are respectively substituted into the visual radar calibration program to construct the attitude residual model and the position residual model. The steps for constructing the attitude residual model and the position residual model include: In the setup phase of the visual radar calibration procedure, a least-squares optimization problem for the visual radar calibration is constructed: Official 1; in, Indicates attitude residual, The observations representing the attitude. This represents the set of variables to be optimized for visual radar. The formula for calculating the observed values ​​representing attitude. The formula for calculating the estimated attitude value. Represents the projection vector; Official 2; in, Indicates the positional residual. Observations representing location, This represents the projection vector of the encoder coordinate system relative to the inertial measurement unit coordinate system. The formula for calculating the observed value of the location is shown below. This represents the state estimate in the direction of rotation. Formulas for calculating estimated location values; Integrating Equation 1 and Equation 2, we obtain the least squares optimization problem: Official 3; in, This represents a least squares optimization problem. Indicates attitude residual, and Represents the weight matrix. Indicates positional residual; The parameter input set, the rotation axis direction, the position and attitude, and the state estimate are substituted into the corresponding positions of Formula 1 and Formula 2 respectively to obtain the attitude residual model and the position residual model; The partial derivatives of the set of variables to be optimized are calculated based on the attitude residual model and the position residual model. The partial derivatives are imported into a preset optimization framework to solve for the rotation axis direction, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder.

2. The calibration method for visual radar as described in claim 1, characterized in that, Before acquiring the relative pose parameters of the camera and inertial measurement unit at various preset angular positions under a preset closed-loop control state, the calibration method includes: Obtain the motor rotor position; The preset closed-loop control state is set based on the motor rotor position.

3. The calibration method for visual radar as described in claim 1, characterized in that, The step of acquiring the relative pose of the camera and the inertial measurement unit at various preset angular positions of the visual radar includes: Obtain camera calibration results and inertial measurement unit calibration results; Based on the camera calibration results and the inertial measurement unit calibration results, the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position are obtained.

4. A calibration system for a visual radar, characterized in that, The calibration system for the visual radar is applied to the calibration method for the visual radar as described in any one of claims 1 to 3, and the calibration system for the visual radar comprises: The sensor calibration module is used to acquire the relative pose parameters of the camera and the inertial measurement unit of the visual radar at various preset angle positions under a preset closed-loop control state. In the preset closed-loop control state, the motor in the visual radar rotates cyclically according to a fixed preset angle position to ensure that the visual radar can accurately acquire the relative pose parameters of the motor at a certain preset angle position. The visual radar calibration module is used to acquire a set of variables to be optimized, consisting of preset angular positions, the rotation axis direction of the visual radar, the position and attitude of the encoder in the visual radar relative to the inertial measurement unit, and the position of the camera relative to the encoder. The state estimate of the set of variables to be optimized is obtained by using a preset state estimation function; Substitute the relative pose parameters and the preset angle positions corresponding to each relative pose parameter into the visual radar calibration program to obtain the parameter input set; The parameter input set, the set of variables to be optimized, and the state estimate are respectively substituted into the visual radar calibration program to construct the attitude residual model and the position residual model. The steps for constructing the attitude residual model and the position residual model include: In the setup phase of the visual radar calibration procedure, a least-squares optimization problem for the visual radar calibration is constructed: Official 1; in, Indicates attitude residual, The observations representing the attitude. This represents the set of variables to be optimized for visual radar. The formula for calculating the observed values ​​representing attitude. The formula for calculating the estimated attitude value. Represents the projection vector; Official 2; in, Indicates the positional residual. Observations representing location, This represents the projection vector of the encoder coordinate system relative to the inertial measurement unit coordinate system. The formula for calculating the observed value of the location is shown below. This represents the state estimate in the direction of rotation. Formulas for calculating estimated location values; Integrating Equation 1 and Equation 2, we obtain the least squares optimization problem: Official 3; in, This represents a least squares optimization problem. Indicates attitude residual, and Represents the weight matrix. Indicates positional residual; The parameter input set, the rotation axis direction, the position and attitude, and the state estimate are substituted into the corresponding positions of Formula 1 and Formula 2 respectively to obtain the attitude residual model and the position residual model; The partial derivatives of the set of variables to be optimized are calculated based on the attitude residual model and the position residual model. The partial derivatives are imported into a preset optimization framework to solve for the rotation axis direction, the position and attitude of the encoder relative to the inertial measurement unit, and the position of the camera relative to the encoder.

5. The calibration system for visual radar as described in claim 4, characterized in that, The sensor calibration module includes a camera calibration module, an inertial measurement unit calibration module, and a camera-inertial measurement unit joint calibration module connected to the output terminals of the camera calibration module and the inertial measurement unit calibration module, respectively. The camera calibration module is used to calibrate the calibration photos transmitted from the visual radar device and generate camera calibration results; The inertial measurement unit calibration module is used to calculate the inertial measurement data transmitted from the visual radar device to obtain the inertial measurement unit calibration result. The camera-inertial measurement unit joint calibration module is used to obtain the relative pose parameters between the camera and the inertial measurement unit in the visual radar at each preset angle position based on the camera calibration results and the inertial measurement unit calibration results.

6. The calibration system for visual radar as described in claim 5, characterized in that, The calibration system of the visual radar establishes a communication connection with the visual radar device. The visual radar device includes a global shutter camera, a micro slip ring, a micro hollow DC brushless motor and a base connected in sequence. The micro slip ring is also connected to the base. The base includes a microcontroller control board, a magnetic field orientation control drive board that is bidirectionally connected to the microcontroller control board, and a processing unit that is bidirectionally connected to both the microcontroller control board and the magnetic field orientation control drive board. The processing unit is also connected to the micro slip ring. The microcontroller control board is equipped with an inertial measurement unit, and the magnetic field orientation control drive board is equipped with an encoder.

7. A visual radar, characterized in that, The visual radar includes: a memory, a processor, and a computer processing program stored in the memory and executable on the processor, the computer processing program being configured to implement the steps of the calibration method for the visual radar as described in any one of claims 1 to 3.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer processing program, which, when executed by a processor, implements the steps of the calibration method for visual radar as described in any one of claims 1 to 3.

Images