Parameter calibration method, parameter calibration device and computer readable storage medium

Abstract

The invention provides a parameter calibration method, a parameter calibration device and a computer readable storage medium. The parameter calibration method comprises the following steps: under the condition that an omnidirectional camera enters a working state, acquiring a scene image by using the omnidirectional camera, and determining an image coordinate of a calibration point for camera calibration in the scene image and a camera coordinate of the calibration point in a coordinate system of the omnidirectional camera; calibrating camera parameters of the omnidirectional camera based on the image coordinates of the calibration point and the camera coordinates, the camera parameters including internal parameters and / or external parameters; and acquiring a real-time image of the target area by using the omnidirectional camera after calibration of the camera parameters. By means of the mode, the parameter calibration device achieves a complete parameter online calibration scheme, and compared with the result depending on offline calibration in the prior art, the parameter calibration method has better precision and stability during online operation.

Description

Technical Field

[0001] The present application relates to the technical field of parameter calibration, and in particular to a parameter calibration method, a parameter calibration method, a parameter calibration device and a computer-readable storage medium. Background Art

[0002] Visual inertial tracking and positioning system is an important underlying technology in the fields of computer vision, robotics, unmanned vehicles, 3D reconstruction and augmented reality. Robust and efficient visual inertial tracking and positioning solutions have important practical value in these fields.

[0003] There are many types of panoramic cameras on the market. Among them, the most common solution for consumer-level panoramic cameras is to use a front- and rear-view binocular omnidirectional camera. How to make a high-precision visual inertial tracking and positioning system based on a binocular omnidirectional camera has also become a current research direction.

[0004] Traditional binocular omnidirectional camera visual inertial tracking and positioning systems use offline calibration parameters and do not have online calibration modules. The parameters of devices from different manufacturers, different production lines, and different batches will vary. If each device is calibrated offline separately, the calibration process will be very complicated. If the differences between devices are not considered and the parameters of all devices are assumed to be the same, the accuracy of tracking and positioning of a single device will be affected. Summary of the invention

[0005] The present application provides a parameter calibration method, a parameter calibration device and a computer-readable storage medium.

[0006] The present application provides a parameter calibration method, which includes:

[0007] When the omnidirectional camera enters a working state, using the omnidirectional camera to capture a scene image, determining image coordinates of calibration points in the scene image for camera calibration, and camera coordinates of the calibration points in a coordinate system of the omnidirectional camera;

[0008] Calibrate camera parameters of the omnidirectional camera based on the image coordinates and camera coordinates of the calibration points, wherein the camera parameters include intrinsic parameters and / or extrinsic parameters;

[0009] The real-time image of the target area is collected using the omnidirectional camera after calibrating the camera parameters.

[0010] In some embodiments, the camera parameters include camera extrinsics, wherein the camera extrinsics include a time deviation between an omnidirectional camera and an inertial sensor;

[0011] The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes:

[0012] Determining the focal length of the omnidirectional camera based on the image coordinates of the calibration points;

[0013] Acquire a first relative rotation relationship between the omnidirectional camera and the inertial sensor, and a second relative rotation relationship between the omnidirectional camera and the world coordinate system;

[0014] Obtaining the world coordinates of the calibration point;

[0015] Get the device's motion speed and angular velocity;

[0016] The time deviation between the omnidirectional camera and the inertial sensor is calibrated based on the device movement speed, the device movement angular velocity, the focal length, the camera coordinates of the calibration point, the first relative rotation relationship, the second relative rotation relationship and the world coordinates.

[0017] In some embodiments, calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes:

[0018] Based on the time deviation between the calibration point and the inertial sensor, the device movement speed, the device movement angular velocity and the device posture, obtaining the compensated device posture of the rolling shutter omnidirectional camera;

[0019] Calibrate a reprojection error term of the omnidirectional camera using the compensation device posture, the world coordinates, and the first relative rotation relationship;

[0020] The time deviation between the omnidirectional camera and the inertial sensor is substituted into the reprojection error term to compensate for the time deviation between the omnidirectional camera and the inertial sensor.

[0021] In some embodiments, the camera parameters include camera internal parameters, wherein the camera internal parameters include the exposure time of each row of the rolling shutter of the omnidirectional camera;

[0022] Obtaining the number of imaging pixel rows of the omnidirectional camera;

[0023] The exposure time of each row of the rolling shutter of the omnidirectional camera is calibrated according to the number of imaging pixel rows and the time deviation.

[0024] In some embodiments, the camera extrinsic parameters include a relative rotation relationship between the omnidirectional camera and the inertial sensor;

[0025] The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes:

[0026] The relative rotation relationship between the omnidirectional camera and the inertial sensor is calibrated based on the focal length, the camera coordinates of the calibration point, the first relative rotation relationship, the second relative rotation relationship, and the world coordinates.

[0027] In some embodiments, the camera extrinsic parameters include a relative position relationship between the omnidirectional camera and the inertial sensor;

[0028] The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes:

[0029] The relative position relationship between the omnidirectional camera and the inertial sensor is calibrated based on the focal length, the camera coordinates of the calibration point, the device motion angular velocity, the time deviation, and the relative rotation relationship.

[0030] In some embodiments, the omnidirectional camera is a binocular omnidirectional camera including a first omnidirectional camera and a second omnidirectional camera; wherein the camera parameters of the first omnidirectional camera are the same as the camera parameters of the second omnidirectional camera.

[0031] In some embodiments, the camera parameters include camera intrinsic parameters, wherein the camera intrinsic parameters include focal length and optical center;

[0032] The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes:

[0033] The focal length of the omnidirectional camera is calibrated by calculating the Jacobian matrix of the focal length using the image coordinates of the calibration points and the camera projection model;

[0034] The Jacobian matrix of the optical center is calculated using the camera coordinates of the calibration point and the camera projection model to calibrate the optical center position of the omnidirectional camera.

[0035] The present application also provides a parameter calibration device, which includes a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the parameter calibration method as described above.

[0036] The present application also provides a computer-readable storage medium, which is used to store program data. When the program data is executed by a processor, it is used to implement the above-mentioned parameter calibration method.

[0037] The beneficial effects of the present application are as follows: when the omnidirectional camera enters the working state, the parameter calibration device uses the omnidirectional camera to collect scene images, determines the image coordinates of the calibration points in the scene image used for camera calibration, and the camera coordinates of the calibration points in the coordinate system of the omnidirectional camera; based on the image coordinates and camera coordinates of the calibration points, calibrates the camera parameters of the omnidirectional camera, wherein the camera parameters include internal parameters and / or external parameters; and uses the omnidirectional camera after calibrating the camera parameters to collect real-time images of the target area. In the above manner, the parameter calibration device realizes a complete online parameter calibration solution. Compared with the existing technology that relies on the results of offline calibration, this parameter calibration method has better accuracy and stability when running online. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for describing the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work. Among them:

[0039] Figure 1 It is a flow chart of an embodiment of a parameter calibration method provided by the present application;

[0040] Figure 2 It is a structural schematic diagram of an embodiment of a parameter calibration device provided by the present application;

[0041] Figure 3 is a structural schematic diagram of another embodiment of the parameter calibration device provided by the present application;

[0042] Figure 4 It is a structural schematic diagram of an embodiment of a computer-readable storage medium provided by the present application. DETAILED DESCRIPTION

[0043] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0044] In order to solve the problems of the above-mentioned binocular omnidirectional camera visual inertial tracking and positioning system, the present application proposes a tracking and positioning solution based on splitting the perspective of the binocular omnidirectional image into several perspective images.

[0045] Please refer to Figure 1 , Figure 1It is a flow chart of an embodiment of a parameter calibration method provided in this application.

[0046] The parameter calibration method of the present application is applied to a parameter calibration device, wherein the parameter calibration device of the present application can be a server or a terminal device, etc., or a system in which a server and a terminal device cooperate with each other. Accordingly, the various parts included in the parameter calibration device, such as various units, subunits, modules, and submodules, can all be set in the server, or can be set in the server and the terminal device respectively.

[0047] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster composed of multiple servers, such as a distributed system, a cloud, etc., or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules used to provide distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the parameter calibration method of the embodiment of the present application can be implemented by a processor calling a computer-readable instruction stored in a memory.

[0048] In a specific embodiment, the parameter calibration device can also be a robot, an unmanned vehicle, a drone, etc., in which case the result of the parameter calibration can be used for visual positioning and navigation. In another specific embodiment, the parameter calibration device can also be an AR device, a VR device, etc., in which case the result of the parameter calibration can be used to enhance the implementation technology, virtual reality technology, etc. The specific product types of the parameter calibration device are not listed here one by one. The parameter calibration device of the present application can also be used as a complete binocular omnidirectional camera visual inertial tracking and positioning system and can be widely used in various scenarios that require visual positioning and navigation, including: autonomous driving, mobile robots, drones, smart phones, virtual reality helmets, augmented reality glasses and other devices that have both binocular omnidirectional image sensors and inertial sensor modules.

[0049] Specifically, if Figure 1 As shown, the parameter calibration method of the embodiment of the present application specifically includes the following steps:

[0050] Step S11: When the omnidirectional camera enters the working state, the omnidirectional camera is used to capture a scene image, and the image coordinates of the calibration points in the scene image for camera calibration and the camera coordinates of the calibration points in the coordinate system of the omnidirectional camera are determined.

[0051] In the embodiment of the present application, the parameter calibration device uses a monocular omnidirectional camera or a binocular omnidirectional camera to collect a calibration image of the target area. Among them, the omnidirectional camera (Omnidirectional Camera Model) is a camera with a 360-degree field of view on the horizontal plane, or a field of view covering a hemisphere or (approximately) the entire sphere.

[0052] Among them, the calibration internal parameters of a monocular omnidirectional camera or a binocular omnidirectional camera can calibrate an ideal pinhole camera model, and can also calibrate various camera models and distortion models, including a pinhole camera model, a spherical camera model, a fisheye camera model, an equidistant distortion model, a radial tangential distortion model, etc.

[0053] It should be noted that, since the front and rear camera modules of the omnidirectional camera are almost completely identical and the camera time has been synchronized by hardware, some of the parameters can be considered to be the same value, thereby sharing a set of calibration parameters. Specifically, the omnidirectional camera may include a first omnidirectional camera and a second omnidirectional camera, wherein the camera parameters of the first omnidirectional camera and the camera parameters of the second omnidirectional camera are the same, that is, the parameter calibration device only needs to calibrate one of the first omnidirectional camera and the second omnidirectional camera.

[0054] It should be noted that the parameter calibration method of the embodiment of the present application is applied in the online image acquisition stage of the omnidirectional camera to realize online automatic calibration, that is, when the parameter calibration device receives a signal that the omnidirectional camera enters the online working state, the parameter calibration device executes the parameter calibration method protected by the present application. It can be understood that the method of the embodiment of the present application can be applied to an uncalibrated omnidirectional camera, or it can also be applied to an omnidirectional camera that has been pre-calibrated offline to obtain offline calibration parameters. In the case of offline calibration, the parameters of the offline calibration can be corrected by the method of the embodiment of the present application. Alternatively, the method of the embodiment of the present application can be applied to any time when the omnidirectional camera is working, and real-time online parameter calibration is performed according to the scene image collected in real time.

[0055] Specifically, the parameter calibration device controls the omnidirectional camera to collect scene images, and presets or places several calibration points in the scene area in advance. Therefore, the parameter calibration device can determine the image coordinates of the calibration points in the scene image for camera calibration, and the camera coordinates of the calibration points in the coordinate system of the omnidirectional camera.

[0056] Step S12: Calibrate the camera parameters of the omnidirectional camera based on the image coordinates and camera coordinates of the calibration points, wherein the camera parameters include intrinsic parameters and / or extrinsic parameters.

[0057] In the embodiment of the present application, the parameter calibration device needs to calibrate the important parameters of the visual inertial tracking and positioning system, mainly including: 1. the internal parameters of each camera; 2. the time offset between each camera and the inertial sensor; 3. the relative position and relative angle between each camera and the inertial sensor; 4. the exposure time of the rolling shutter of each camera.

[0058] Step S13: using the omnidirectional camera with calibrated camera parameters to capture a real-time image of the target area.

[0059] In the embodiment of the present application, after the parameter calibration device completes the online calibration of the omnidirectional camera, it can start controlling the omnidirectional camera to start working.

[0060] It should be noted that if the visual inertial tracking and positioning system uses the inertial sensor as the system coordinate system, the relative position and posture of each camera relative to the inertial sensor is calibrated online. If the visual inertial tracking and positioning system uses a binocular camera as the system coordinate system, the relative position and posture of the binocular camera and the relative position and posture of the inertial sensor and the binocular camera are calibrated online.

[0061] The following is an introduction to the symbols used in the parameter calibration scheme:

[0062] In this scheme, the inertial navigation coordinate system is taken as the device's own coordinate system, and the displacement of the device can be expressed as G p B , the rotation can be expressed as G R B .

[0063] It should be noted that the device is a device equipped with an omnidirectional camera and an inertial sensor, such as but not limited to: autonomous vehicles, mobile robots, drones, smart phones, virtual reality helmets, augmented reality glasses or other devices with both binocular omnidirectional image sensors and inertial sensor modules.

[0064] In the scheme, the relative position between a camera and the inertial sensor is B p C , the relative rotation is B R C .

[0065] The camera model in the scheme is set as an ideal pinhole camera model, and its optical centers are c x and c y , the focal lengths are f x and f y , where x and y are the horizontal and vertical axes of the vacuum camera model respectively.

[0066] The exposure time of each row of the rolling shutter is set to t r , the time deviation between the camera and the inertial sensor is td , a single frame image has N rows, and the current feature point p f At row n.

[0067] The following is a description of the calibration process of the important parameters of the visual inertial tracking and positioning system combined with the above symbol descriptions:

[0068] Traditional methods do not take into account that the external parameters between the camera and the inertial sensor will change. The device will inevitably be bumped or dropped during use, causing the camera lens to rotate or shift. This application proposes an online calibration scheme for camera external parameters, which can estimate the external parameter information between each camera and the inertial sensor in real time.

[0069] Compared with monocular omnidirectional cameras, binocular omnidirectional cameras have some unique features. If there is a large difference between the calibration value and the actual value of the external parameters between binocular cameras, it will cause a conflict between the visual projection systems of the two, which will further deteriorate the accuracy of the algorithm. Traditional methods do not take into account the impact of this external parameter conflict on the system. This application calibrates the external parameter information between the binocular camera and the inertial sensor respectively, which significantly improves the system accuracy.

[0070] The inertial sensor and camera of the device are mostly independently timed, without hardware synchronization. If the time of the two is not synchronized for a long time, the difference between their time systems will become larger and larger. Therefore, the parameter calibration device can obtain the current feature point p according to the above symbols f The time deviation t between the exposure time point and the inertial system δ for:

[0071]

[0072] In the embodiment of the present application, the device motion speed in the current inertial sensor coordinate system is assumed to be v b , the angular velocity is w b , then the corresponding feature point p f For , the corresponding device posture is:

[0073]

[0074] In the embodiment of the present application, let point m = (u, v) T Normalized two-dimensional point in the camera coordinate system, I = (i x ,i y ) T For a two-dimensional point on the image, there is an image coordinate calculation formula:

[0075]

[0076] set up c p f=(x, y, z) T is a three-dimensional point in the camera coordinate system, and there is a normalization formula:

[0077]

[0078] The position of the 3D point in the world coordinate system is known G p f , we can get C p f The transformation formula is:

[0079]

[0080] In the embodiment of the present application, the above formulas can be combined to obtain the reprojection error term formula of the three-dimensional point:

[0081]

[0082]

[0083] According to formula (7), the time deviation of the rolling shutter can be added to the reprojection error term, so that the system can compensate for the time deviation of the rolling shutter online in real time.

[0084] Among them, the reprojection error refers to the difference between the projection of the real 3D space point on the image plane (that is, the pixel point on the image) and the reprojection (actually the virtual pixel point obtained by our calculation value). Due to various reasons, the calculated value will not be completely consistent with the actual situation, that is, this difference cannot be exactly 0. At this time, it is necessary to minimize the sum of these differences to obtain the optimal camera pose parameters and the coordinates of the 3D space point.

[0085] Reprojection error is often used in computer vision. For example, when calculating the homography matrix and projection matrix, the cost function is often constructed using the reprojection error, and then the cost function is minimized to optimize the homography matrix or projection matrix. The reason for using the reprojection error is that it not only takes into account the calculation error of the homography matrix, but also the measurement error of the image points, so its accuracy will be higher.

[0086] Specifically, the camera parameters of the present application include camera intrinsic parameters, wherein the camera intrinsic parameters include focal length and optical center.

[0087] The focal length calibration process is as follows:

[0088] f x The Jacobian matrix of :

[0089]

[0090] fy The Jacobian matrix of :

[0091]

[0092] The parameter calibration device determines the focal length (f x , f y ).

[0093] The calibration process of the optical center is as follows:

[0094] c x The Jacobian matrix of :

[0095]

[0096] c y The Jacobian matrix of :

[0097]

[0098] The time deviation t between the omnidirectional camera and the inertial sensor d The calibration process is as follows:

[0099] Based on the image coordinates (u, v) of the calibration point, determine the first exposure matrix J k ; Based on the camera coordinates (x, y, z) of the calibration point, determine the second exposure matrix J n ; Obtain the first relative rotation relationship between the omnidirectional camera and the inertial sensor B R C , and the second relative rotation relationship between the omnidirectional camera and the world coordinate system B R G ; Get the world coordinates of the calibration point G p f ; Obtain the device motion speed v and the device motion angular velocity ω; Based on the device motion speed v, the device motion angular velocity ω, and the first exposure matrix J k , the second exposure matrix J n , the first relative rotation relationship B R C The second relative rotation relationship B R G And the world coordinates G p f , calibrate the time deviation t between the omnidirectional camera and the inertial sensor d .

[0100] The parameter calibration device calibrates the time deviation between the omnidirectional camera and the inertial sensor based on the device movement speed v, the device movement angular speed ω, the focal length, the camera coordinates of the calibration point, the first relative rotation relationship, the second relative rotation relationship and the world coordinates.

[0101] Specifically, the time deviation t between the omnidirectional camera and the inertial sensor is d The Jacobian matrix is ​​as follows:

[0102]

[0103] in:

[0104]

[0105] Furthermore, the parameter calibration device can also use the above time deviation t d Calibrate the exposure time t of each row of the rolling shutter of the omnidirectional camera r .

[0106] The exposure time t of each row of the rolling shutter of the omnidirectional camera r The Jacobian matrix is ​​as follows:

[0107]

[0108] Furthermore, the camera external parameters calibrated in the embodiment of the present application also include the relative rotation relationship between the omnidirectional camera and the inertial sensor. B R C Specifically, the parameter calibration device is based on the first exposure matrix J k , the second exposure matrix H n , the first relative rotation relationship B R C The second relative rotation relationship B R G And the world coordinates G p f , calibrate the relative rotation relationship between the omnidirectional camera and the inertial sensor B R C .

[0109] Relative rotation relationship between omnidirectional camera and inertial sensor B R C The Jacobian matrix is ​​as follows:

[0110]

[0111] Furthermore, the camera external parameters calibrated in the embodiment of the present application also include the relative position relationship between the omnidirectional camera and the inertial sensor. B pC Based on the first exposure matrix J k , the second exposure matrix J n , the device movement angular velocity ω, the time deviation t δ And the relative rotation relationship B R C , calibrate the relative position relationship between the omnidirectional camera and the inertial sensor B p C .

[0112] The relative position relationship between the omnidirectional camera and the inertial sensor B p C The Jacobian matrix is ​​as follows:

[0113]

[0114] The Jacobian matrix of the calibration parameters represents the linearized gradient direction, and a variety of optimization algorithms can be used to solve the system, where the optimizers are based on the Jacobian matrix and the Hessian matrix derived from the Jacobian. Specifically, a variety of optimization algorithms can be used to solve the online calibration parameters, including extended Kalman filter, Gauss-Newton optimizer, gradient descent optimization algorithm, Levenberg optimization algorithm, etc.

[0115] When the omnidirectional camera enters the working state, the parameter calibration device uses the omnidirectional camera to collect scene images, determine the image coordinates of the calibration points in the scene image used for camera calibration, and the camera coordinates of the calibration points in the coordinate system of the omnidirectional camera; based on the image coordinates and camera coordinates of the calibration points, calibrate the camera parameters of the omnidirectional camera, wherein the camera parameters include internal parameters and / or external parameters; and use the omnidirectional camera after calibrating the camera parameters to collect real-time images of the target area. In the above manner, the parameter calibration device realizes a complete online parameter calibration solution. Compared with the existing technology that relies on the results of offline calibration, this parameter calibration method has better accuracy and stability when running online.

[0116] The parameter calibration method of the embodiment of the present application does not need to perform detailed offline calibration on each device, eliminating a large number of complicated offline calibration processes; if there is a large difference between the offline calibration parameters and the actual parameters, the offline parameters can be conveniently corrected to improve the accuracy and stability of the system; due to changes in system parameters caused by various uncontrollable factors, the system can quickly adapt to the changes and still be able to output stable tracking and positioning results.

[0117] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0118] In order to implement the parameter calibration method of the above embodiment, the present application also proposes a parameter calibration device, for details, please refer to Figure 2 , Figure 2 It is a structural schematic diagram of an embodiment of a parameter calibration device provided in this application.

[0119] The parameter calibration device 400 of the embodiment of the present application includes a collection module 41 , a calibration module 42 , and a working module 43 .

[0120] Among them, the acquisition module 41 is used to use the omnidirectional camera to acquire scene images when the omnidirectional camera enters the working state, determine the image coordinates of the calibration points in the scene image used for camera calibration, and the camera coordinates of the calibration points in the coordinate system of the omnidirectional camera.

[0121] The calibration module 42 is used to calibrate the camera parameters of the omnidirectional camera based on the image coordinates and camera coordinates of the calibration points, wherein the camera parameters include intrinsic parameters and / or extrinsic parameters.

[0122] The working module 43 is used to collect a real-time image of the target area using the omnidirectional camera after calibrating the camera parameters.

[0123] In order to implement the parameter calibration method of the above embodiment, this application also proposes another parameter calibration device, please refer to Figure 3 , Figure 3 It is a structural schematic diagram of another embodiment of the parameter calibration device provided in the present application.

[0124] The parameter calibration device 500 of the embodiment of the present application includes a memory 51 and a processor 52, wherein the memory 51 and the processor 52 are coupled.

[0125] The memory 51 is used to store program data, and the processor 52 is used to execute the program data to implement the parameter calibration method described in the above embodiment.

[0126] In this embodiment, the processor 52 may also be referred to as a CPU (Central Processing Unit). The processor 52 may be an integrated circuit chip having the ability to process signals. The processor 52 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor may be a microprocessor or the processor 52 may also be any conventional processor, etc.

[0127] To implement the parameter calibration method of the above embodiment, the present application also provides a computer-readable storage medium, such as Figure 4 As shown, the computer-readable storage medium 600 is used to store program data 61. When the program data 61 is executed by the processor, it is used to implement the parameter calibration method described in the above embodiment.

[0128] The present application also provides a computer program product, wherein the computer program product includes a computer program, and the computer program is operable to enable a computer to execute the parameter calibration method as described in the embodiment of the present application. The computer program product may be a software installation package.

[0129] The parameter calibration method described in the above embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or part of the contribution to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) or a processor (processor) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0130] The above description is only an implementation method of the present application, and does not limit the patent scope of the present application. Any equivalent structure or equivalent process transformation made using the contents of the present application specification and drawings, or directly or indirectly used in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A parameter calibration method, characterized in that: The parameter calibration method comprises: When the omnidirectional camera enters a working state, using the omnidirectional camera to capture a scene image, determining image coordinates of calibration points in the scene image for camera calibration, and camera coordinates of the calibration points in a coordinate system of the omnidirectional camera; Calibrate camera parameters of the omnidirectional camera based on the image coordinates and camera coordinates of the calibration points, wherein the camera parameters include intrinsic parameters and / or extrinsic parameters; The real-time image of the target area is collected using the omnidirectional camera after calibrating the camera parameters.

2. The parameter calibration method according to claim 1, characterized in that: The camera parameters include camera extrinsics, wherein the camera extrinsics include a time deviation between an omnidirectional camera and an inertial sensor; The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes: Determining the focal length of the omnidirectional camera based on the image coordinates of the calibration points; Acquire a first relative rotation relationship between the omnidirectional camera and the inertial sensor, and a second relative rotation relationship between the omnidirectional camera and the world coordinate system; Obtaining the world coordinates of the calibration point; Get the device's motion speed and angular velocity; The time deviation between the omnidirectional camera and the inertial sensor is calibrated based on the device movement speed, the device movement angular velocity, the focal length, the camera coordinates of the calibration point, the first relative rotation relationship, the second relative rotation relationship and the world coordinates.

3. The parameter calibration method according to claim 2, characterized in that: The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes: Based on the time deviation between the calibration point and the inertial sensor, the device movement speed, the device movement angular velocity and the device posture, obtaining the compensated device posture of the rolling shutter omnidirectional camera; Calibrate a reprojection error term of the omnidirectional camera using the compensation device posture, the world coordinates, and the first relative rotation relationship; The time deviation between the omnidirectional camera and the inertial sensor is substituted into the reprojection error term to compensate for the time deviation between the omnidirectional camera and the inertial sensor.

4. The parameter calibration method according to claim 2, characterized in that: The camera parameters include camera internal parameters, wherein the camera internal parameters include the exposure time of each row of the rolling shutter of the omnidirectional camera; Obtaining the number of imaging pixel rows of the omnidirectional camera; The exposure time of each row of the rolling shutter of the omnidirectional camera is calibrated according to the number of imaging pixel rows and the time deviation.

5. The parameter calibration method according to claim 2, characterized in that: The camera external parameters include the relative rotation relationship between the omnidirectional camera and the inertial sensor; The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes: The relative rotation relationship between the omnidirectional camera and the inertial sensor is calibrated based on the focal length, the camera coordinates of the calibration point, the first relative rotation relationship, the second relative rotation relationship, and the world coordinates.

6. The parameter calibration method according to claim 5, characterized in that: The camera external parameters include the relative position relationship between the omnidirectional camera and the inertial sensor; The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes: The relative position relationship between the omnidirectional camera and the inertial sensor is calibrated based on the focal length, the camera coordinates of the calibration point, the angular velocity of the device movement, the time deviation and the relative rotation relationship.

7. The parameter calibration method according to claim 1, characterized in that: The omnidirectional camera is a binocular omnidirectional camera including a first omnidirectional camera and a second omnidirectional camera; wherein the camera parameters of the first omnidirectional camera are the same as the camera parameters of the second omnidirectional camera.

8. The parameter calibration method according to claim 1, characterized in that: The camera parameters include camera intrinsic parameters, wherein the camera intrinsic parameters include focal length and optical center; The step of calibrating the camera parameters of the omnidirectional camera based on the image coordinates and the camera coordinates of the calibration points includes: The focal length of the omnidirectional camera is calibrated by calculating the Jacobian matrix of the focal length using the image coordinates of the calibration points and the camera projection model; The Jacobian matrix of the optical center is calculated using the camera coordinates of the calibration point and the camera projection model to calibrate the optical center position of the omnidirectional camera.

9. A parameter calibration device, characterized in that: The parameter calibration device comprises a processor and a memory, wherein program data is stored in the memory, and the processor is used to execute the program data to implement the parameter calibration method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store program data, and when the program data is executed by a processor, it is used to implement the parameter calibration method according to any one of claims 1 to 8.

Images