Method and device for calibrating camera device

Abstract

The present invention provides a correction method and device for a camera device, wherein a plurality of cameras are used to locate a target, and each camera device is provided with a corresponding inertial measurement unit, including: obtaining the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data represents the posture of the camera device equivalent to a calibration plate, and the inertial data includes acceleration data and angular velocity data collected by the inertial measurement unit; based on the posture data and inertial data, determining the transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device; and correcting the camera device based on the current inertial data and the transformation matrix collected by the inertial measurement unit. The camera device can be calibrated and corrected by the inertial measurement unit, and the posture of the camera device can be monitored in real time without manual operation, and the offset of the camera device can be measured online, so that changes in the posture of the camera device can be discovered in a timely manner.

Description

Technical Field

[0001] The present invention relates to the technical field of camera calibration, and in particular to a calibration method and device for a camera device. Background Art

[0002] The production and inspection of large equipment in welding workshops, painting workshops, and assembly workshops require these large equipment to be positioned in 6 degrees of freedom in space during the operation of the production line. In the prior art, multiple 2D cameras are generally used to extract the surface features of the workpiece to correct the body posture. For example, four 2D cameras can be used for body positioning. After calibration, in a complex production environment, there are uncertain external factors that cause the posture of these calibrated cameras to change, such as collisions during debugging.

[0003] However, changes in the camera's posture can cause large errors or mistakes in the vehicle body positioning system. Existing technologies are unable to measure the camera's offset online and monitor the camera's posture in real time, and problems can only be discovered when the entire system has problems. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a correction method and device for a camera device, so as to calibrate and correct the camera device through an inertial measurement unit, monitor the posture of the camera device in real time without human operation, measure the offset of the camera device online, and promptly detect changes in the posture of the camera device.

[0005] In a first aspect, an embodiment of the present invention provides a calibration method for a camera device, wherein multiple camera devices are used to locate a target, and each camera device is provided with a corresponding inertial measurement unit, and the method comprises: acquiring posture data collected by each camera device and inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data characterizes the posture of the camera device equivalent to a calibration plate, and the inertial data comprises acceleration data and angular velocity data of the inertial measurement unit; based on the posture data and the inertial data, determining a transformation matrix between a coordinate system of each inertial measurement unit and a coordinate system of the corresponding camera device; and calibrating the camera device based on current inertial data and the transformation matrix collected by the inertial measurement unit.

[0006] In an optional embodiment of the present application, the above-mentioned step of obtaining the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device includes: forming a positioning device with the camera device and the corresponding inertial measurement unit, and setting the positioning device in front of the calibration plate; moving the positioning device within a preset range through an external mechanical device, and determining that the field of view of the camera device includes the calibration plate during the movement of the positioning device; obtaining the posture data collected by each camera device and the inertial measurement unit corresponding to each camera device during the movement of the positioning device.

[0007] In an optional embodiment of the present application, the above-mentioned step of obtaining the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device includes: sending a signal to each camera device and the corresponding inertial measurement unit through an external trigger; each camera device collects posture data based on the signal; and the inertial measurement unit corresponding to each camera device collects inertial data based on the signal.

[0008] In an optional embodiment of the present application, the acquisition frequency of the above-mentioned camera device is lower than the acquisition frequency of the inertial measurement unit, and the method further includes: aligning the posture data with the inertial data by interpolation.

[0009] In an optional embodiment of the present application, the step of aligning the posture data with the inertial data by interpolation includes: smoothing the inertial data; wherein the smoothing includes: Kalman filtering; determining the time delay value between the posture data and the inertial data based on the time series of the posture data and the time series of the inertial data; and aligning the posture data with the smoothed inertial data by interpolation based on the time delay value.

[0010] In an optional embodiment of the present application, the above-mentioned step of correcting the camera device based on the current inertial data and the transformation matrix collected by the inertial measurement unit includes: if the change between the current inertial data collected by the inertial measurement unit and the historical inertial data collected at the previous moment is greater than a preset threshold, determining that the posture of the camera device has changed; if it is determined that the posture of the camera device has changed, determining the posture change data of the camera device based on the change and the transformation matrix; determining the offset of the camera device based on the posture change data of the camera device, and correcting the camera device based on the offset.

[0011] In an optional embodiment of the present application, after the above-mentioned step of determining the posture change data of the camera device based on the change amount and the transformation matrix, the method also includes: acquiring the historical posture data of the camera device; and determining the current posture data of the camera device based on the posture change data and the historical posture data.

[0012] In an optional embodiment of the present application, the above-mentioned target includes a vehicle, and multiple camera devices locate the vehicle by capturing surface features of the vehicle.

[0013] In an optional embodiment of the present application, the above-mentioned imaging device includes: a camera or a lidar.

[0014] In a second aspect, an embodiment of the present invention further provides a correction device for a camera device, wherein multiple camera devices are used to locate a target, and each camera device is provided with a corresponding inertial measurement unit, and the device comprises: a data acquisition module, used to obtain posture data collected by each camera device, and inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data characterizes the posture of the camera device equivalent to the calibration plate, and the inertial data comprises acceleration data and angular velocity data of the inertial measurement unit; a transformation matrix determination module, used to determine the transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device based on the posture data and the inertial data; and a camera device correction module, used to calibrate the camera device based on the current inertial data and the transformation matrix collected by the inertial measurement unit.

[0015] The embodiments of the present invention bring the following beneficial effects:

[0016] The embodiment of the present invention provides a correction method and device for a camera device, which obtains the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes the acceleration data and angular velocity data of the inertial measurement unit; based on the posture data and the inertial data, the transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device is determined; based on the current inertial data and the transformation matrix collected by the inertial measurement unit, the camera device is corrected. In this way, the camera device can be calibrated and corrected by the inertial measurement unit, the posture of the camera device can be monitored in real time without manual operation, the offset of the camera device can be measured online, and the change of the posture of the camera device can be discovered in time.

[0017] Other features and advantages of the present disclosure will be set forth in the following description, or some features and advantages may be inferred or unambiguously determined from the description, or may be learned by implementing the above-mentioned technology of the present disclosure.

[0018] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are specifically cited below and described in detail with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 A flowchart of a calibration method for a camera device provided by an embodiment of the present invention;

[0021] Figure 2 A flowchart of another method for calibrating a camera device provided by an embodiment of the present invention;

[0022] Figure 3 A schematic diagram of a sampling time and a timestamp generation time provided by an embodiment of the present invention;

[0023] Figure 4 A schematic diagram of smoothing inertial data provided by an embodiment of the present invention;

[0024] Figure 5 A schematic structural diagram of a correction device for a camera device provided by an embodiment of the present invention;

[0025] Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] At present, the production and inspection of large equipment such as welding workshops, painting workshops, and assembly workshops require these large equipment to be positioned in 6 degrees of freedom in space during the operation of the production line. In the prior art, multiple 2D cameras are generally used to extract the surface features of the workpiece to correct the body posture. For example, four 2D cameras can be used for body positioning. After calibration, in a complex production environment, there are uncertain external factors that cause the posture of these calibrated cameras to change, such as collisions during debugging.

[0028] However, changes in the camera's posture can cause large errors or mistakes in the vehicle body positioning system. Existing technologies are unable to measure the camera's offset online and monitor the camera's posture in real time, and problems can only be discovered when the entire system has problems.

[0029] Based on this, the embodiment of the present invention provides a calibration method and device for a camera device, specifically providing an automatic calibration method for a vehicle body positioning camera without manual operation, which can utilize an inertial measurement unit (IMU) and a camera, improve the optimization algorithm, simplify the calibration process, and thus achieve:

[0030] (1) The camera posture is monitored in real time through IMU. When the camera posture suddenly changes significantly, it can respond online and be timely.

[0031] (2) The camera offset is compensated through the inertial data of the IMU, so that the camera offset can be corrected without human operation.

[0032] (3) After adding a new vehicle model, if the camera posture needs to be changed, there is no need to calibrate again. The camera posture can be directly adjusted and the adjusted camera posture can be calculated through the algorithm.

[0033] To facilitate understanding of this embodiment, a calibration method for a camera device disclosed in an embodiment of the present invention is first introduced in detail.

[0034] Embodiment 1:

[0035] An embodiment of the present invention provides a calibration method for a camera device, wherein a plurality of camera devices are used to locate a target, and each camera device is provided with a corresponding inertial measurement unit.

[0036] In this embodiment, a target can be positioned by multiple camera devices, for example, a vehicle can be positioned by multiple cameras. In this embodiment, a corresponding inertial measurement unit is provided for each camera device, for example, a corresponding inertial measurement unit is provided above each camera.

[0037] Based on the above description, see Figure 1 A flow chart of a calibration method for a camera device is shown, and the calibration method for a camera device comprises the following steps:

[0038] Step S102 , acquiring the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device.

[0039] The posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes acceleration data and angular velocity data of the inertial measurement unit. In this embodiment, each camera device can be combined with a corresponding inertial measurement unit to collect data together. The camera device collects the posture of the camera device equivalent to the calibration plate as the posture data, and the corresponding inertial measurement unit collects the acceleration data and angular velocity data of the inertial measurement unit as the inertial data.

[0040] Step S104 : determining a transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device based on the posture data and the inertial data.

[0041] This embodiment can determine the transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device based on the posture data and the inertial data. It should be noted that in this embodiment, corresponding inertial measurement units are set for each of the N camera devices, and the transformation matrix between the coordinate system of the N inertial measurement units and the coordinate system of the corresponding camera device can be obtained.

[0042] For example, assuming that four cameras are provided as imaging devices, four inertial measurement units need to be provided for the four cameras respectively, and the transformation matrix between the coordinate systems of the four inertial measurement units and the coordinate system of the corresponding imaging devices needs to be calculated.

[0043] Step S106: calibrate the camera device based on the current inertial data collected by the inertial measurement unit and the transformation matrix.

[0044] After the transformation matrix is ​​determined, this embodiment can calibrate the camera device based on the current inertial data collected by the inertial measurement unit and the transformation matrix.

[0045] For example, when the camera moves, the current inertial data collected by the inertial measurement unit can determine that the camera moves. Then, the displacement of the camera can be determined based on the current inertial data and the transformation matrix, so as to calibrate the camera.

[0046] In summary, in this embodiment, a transformation matrix can be first determined by the posture data of the camera device and the inertial data of the IMU for calibration. When the camera device and the IMU are actually placed on site, the inertial data of the IMU can be used to determine whether the camera device has excessive displacement. If excessive displacement occurs, the camera device can be corrected based on the inertial data of the IMU and the transformation matrix obtained by calibration.

[0047] The embodiment of the present invention provides a correction method for a camera device, which obtains the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes the acceleration data and angular velocity data of the inertial measurement unit; based on the posture data and the inertial data, the transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device is determined; based on the current inertial data collected by the inertial measurement unit and the transformation matrix, the camera device is corrected. In this method, the camera device can be calibrated and corrected by the inertial measurement unit, the posture of the camera device can be monitored in real time without manual operation, the offset of the camera device can be measured online, and the change of the posture of the camera device can be discovered in time.

[0048] Embodiment 2:

[0049] This embodiment provides another method for calibrating an imaging device, which is implemented on the basis of the above embodiment. In some embodiments, the imaging device includes: a camera or a laser radar. In this embodiment, a camera or a laser radar can be used as the imaging device to locate the target through the camera or the laser radar.

[0050] In some embodiments, the target includes a vehicle, and the multiple cameras locate the vehicle by collecting surface features of the vehicle. In this embodiment, the vehicle can be used as the target, and the multiple cameras can locate the vehicle. Specifically, the multiple cameras can locate the vehicle by collecting surface features of the vehicle.

[0051] It should be noted here that this embodiment will be described later using multiple cameras to locate the vehicle as an example, which will not be repeated hereafter.

[0052] Based on the above description, see Figure 2 FIG. 1 is a flow chart of another calibration method for a camera device shown in FIG. 1 . The calibration method for a camera device in this embodiment includes the following steps:

[0053] Step S202, the camera device and the corresponding inertial measurement unit form a positioning device, and the positioning device is set in front of the calibration plate; the positioning device is moved within a preset range by an external mechanical device, and it is determined that the field of view of the camera device includes the calibration plate during the movement of the positioning device.

[0054] Among them, the camera device and the corresponding inertial measurement unit are rigidly connected to ensure that the two remain relatively still during the movement of the positioning device.

[0055] This embodiment involves the coordinate system from the IMU To the camera's coordinate system The fixed transformation of represents the coordinate system of the calibration plate. The relative position of any two coordinate systems can be expressed as The homogeneous matrix Indicates, for example about The relationship can be obtained by transforming the matrix express, This represents a mapping of points expressed in homogeneous coordinates: ; ;

[0056] in, represents the rotation matrix, Indicates from Origin to The problem is to calculate the coordinate system of the IMU through a batch of images and the acceleration data and gyroscope data of the IMU. To the camera's coordinate system Fixed transformation of The maximum likelihood estimate (MLE) of .

[0057] The positioning device can be composed of a calibrated global shutter camera and an IMU rigidly connected, and moved in front of a fixed visual calibration target (i.e., a calibration plate). The positioning device is moved within a certain range using an external mechanical device (such as a manipulator, a turntable, etc.) to ensure that the pattern of the calibration plate is clearly captured in the camera's field of view.

[0058] Step S204 , acquiring the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device during the movement of the positioning device.

[0059] In this embodiment, the posture data (image) collected by the camera and the IMU data (i.e., inertial data, including acceleration data and angular velocity data) collected by the IMU can be recorded as measurement values ​​during the movement of the positioning device, and the motion trajectory information estimated by the positioning device data can be used as the current state of the positioning device.

[0060] In some embodiments, a signal may be sent to each camera device and a corresponding inertial measurement unit through an external trigger; each camera device collects posture data based on the signal; and the inertial measurement unit corresponding to each camera device collects inertial data based on the signal.

[0061] See also Figure 3A schematic diagram of a sampling time and a timestamp generation time is shown. In this embodiment, an external trigger can be used to simultaneously trigger the camera to capture attitude data and the IMU to collect inertial data with the same signal, providing a more accurate timestamp at the hardware level. The external trigger can be a PLC (Programmable Logic Controller).

[0062] Step S206 , determining a transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device based on the posture data and the inertial data.

[0063] In some embodiments, the acquisition frequency of the camera device is lower than the acquisition frequency of the inertial measurement unit, and the posture data and the inertial data can be aligned by interpolation.

[0064] Considering that there is still a time delay at the data reading and writing level, and the acquisition frequency of IMU and camera is very different, the acquisition frequency of IMU is much higher than that of camera. The reason why there is a big difference between the acquisition frequency of IMU and camera is: 1. Generally, cameras can collect dozens of frames of images per second, so the acquisition frequency is tens of hertz. IMU will collect acceleration information hundreds to thousands of times per second, so the acquisition frequency of IMU is much higher than that of camera. 2. Delay at the reading and writing level: From data acquisition to data writing to register, the camera data must first be transmitted through its own core operation, and IMU must also go through its own calculation first. In theory, there is a difference in the calculation speed of the two.

[0065] Therefore, this embodiment can use the interpolation algorithm to accurately align the data later. There is a time difference between the camera and the IMU. . It can be expressed as .

[0066] Among them, the posture data collected by the camera can be The pose of each camera relative to the calibration plate is expressed by Transformation Matrix The IMU data is the IMU acceleration and angular velocity data corresponding to the attitude data.

[0067] In some embodiments, the inertial data can be smoothed; wherein the smoothing includes: Kalman filtering; determining the time delay value between the posture data and the inertial data based on the time series of the posture data and the time series of the inertial data; and aligning the posture data with the smoothed inertial data by interpolation based on the time delay value.

[0068] For data processing, the Kalman filter can be used to smooth the inertial data of the IMU. The following operations are performed on the data of each time step:

[0069] Estimation steps: ; ;

[0070] in, is the state estimate at time k, is the state transition matrix, is the input control matrix, is the input control vector, is the estimated covariance matrix, is the process noise covariance matrix.

[0071] Update steps: ; ; ;

[0072] in, is the Kalman gain, is the observation matrix, is the measurement noise covariance matrix, is the measured value. In practice, it is necessary to select appropriate filter parameters according to the specific application scenario and performance requirements. The Kalman filter will update the state according to the current observation value to achieve smooth data.

[0073] See also Figure 4 A schematic diagram of smoothing inertial data is shown in FIG. Figure 4 2 shows the acceleration data before and after filtering. It can be seen that the filtered inertial data is smoother, thereby reducing the impact of noise.

[0074] For data alignment, the time offset corresponding to the maximum correlation between attitude data and inertial data can be found: ;

[0075] in, and represents the time series of inertial data and attitude data, Represents the correlation between the two at time t. Calculate the peak of the cross-correlation function, corresponding to the time delay between the IMU and the camera, which can be expressed as: ; This method is used to align the IMU and camera data.

[0076] For interpolation, the smoothed inertial data can be represented using quaternions, based on the timestamp of the image. Find the nearest IMU timestamp and Then perform Slerp difference: ; ;

[0077] Compute fixed transformation Through the above calculation, we can get multiple sets of aligned camera posture matrices. Represented by, and the IMU attitude matrix, express: .

[0078] It can be expressed as a problem of minimizing the loss function, ,in is the Frobenius norm, calculate the derivative of the loss function and set it to zero to find the optimal solution : ; ;make , use the gradient descent method to iteratively solve the optimal solution . It is the relative pose between IMU and camera.

[0079] Step S208: calibrate the camera device based on the current inertial data collected by the inertial measurement unit and the transformation matrix.

[0080] In some embodiments, if the change between the current inertial data collected by the inertial measurement unit and the historical inertial data collected at the previous moment is greater than a preset threshold, it is determined that the posture of the camera device has changed; if it is determined that the posture of the camera device has changed, the posture change data of the camera device is determined based on the change and the transformation matrix; the offset of the camera device is determined based on the posture change data of the camera device, and the camera device is corrected based on the offset.

[0081] In this embodiment, a threshold of inertial data can be pre-set to calculate the change between the current inertial data collected by the inertial measurement unit at the current moment and the historical inertial data collected at the previous moment. If the change is greater than the threshold, it can be considered that the inertial data has a sudden change, and it is determined that the posture of the camera has changed, thereby realizing online detection of the posture of the camera, and responding in time when the posture of the camera changes. Among them, the first-order derivative of the data collected by the IMU can be used as the change between the current inertial data and the historical inertial data collected at the previous moment.

[0082] In this embodiment, the offset of the attitude camera can be calculated by the IMU data. The change between the current inertial data and the historical inertial data of the IMU is expressed as describe, That is the camera's posture change data, where is the historical posture data of the camera, is the transformation matrix.

[0083] Therefore, this embodiment can compensate for the offset of the camera through the inertial data of the IMU, thereby correcting the camera offset without human operation.

[0084] In some embodiments, historical posture data of the camera device may be acquired; and current posture data of the camera device may be determined based on the posture change data and the historical posture data.

[0085] After determining the posture change data of the camera, the present embodiment may also determine the historical posture data of the camera device, and determine the current posture data of the camera according to the posture change data and the historical posture data.

[0086] In the prior art, when using a camera to locate a vehicle, if the vehicle model is changed, the position of the camera needs to be changed to capture the features of the newly added vehicle model, so the camera needs to be recalibrated. However, this embodiment can calculate the adjusted camera posture data by using the inertial data and transformation matrix of the IMU, without the need for recalibration.

[0087] In addition, the performance evaluation index of the solution provided in this embodiment can be calculated by the mean absolute error (MAE) and the mean absolute percentage error (MAPE). MAE is the average value of the absolute error between the actual value and the calculated value, and the formula is as follows: ;

[0088] MAPE is the average absolute percentage error between the actual value and the calculated value, and the formula is as follows: .

[0089] Among them, the above performance evaluation indicators can be used to determine whether the calibration accuracy meets the requirements. The calibration results and real data are substituted into the above two formulas to obtain MAE and MAPE. If both MAE and MAPE are less than the preset threshold, it can be considered that the accuracy meets the requirements.

[0090] Embodiment three:

[0091] Corresponding to the above method embodiment, the embodiment of the present invention provides a calibration device for a camera device, wherein a plurality of camera devices are used to locate a target, and each camera device is provided with a corresponding inertial measurement unit. Figure 5 A structural schematic diagram of a correction device for a camera device is shown, and the correction device for the camera device includes:

[0092] The data acquisition module 61 is used to obtain the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes the acceleration data and angular velocity data of the inertial measurement unit;

[0093] A transformation matrix determination module 62, for determining a transformation matrix between a coordinate system of each inertial measurement unit and a coordinate system of a corresponding camera device based on the posture data and the inertial data;

[0094] The camera device calibration module 63 is used to calibrate the camera device based on the current inertial data and the transformation matrix collected by the inertial measurement unit.

[0095] The embodiment of the present invention provides a correction device for a camera device, which obtains the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device; the posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes the acceleration data and angular velocity data of the inertial measurement unit; based on the posture data and the inertial data, the transformation matrix between the coordinate system of each inertial measurement unit and the coordinate system of the corresponding camera device is determined; based on the current inertial data collected by the inertial measurement unit and the transformation matrix, the camera device is corrected. In this way, the camera device can be calibrated and corrected by the inertial measurement unit, the posture of the camera device can be monitored in real time without manual operation, the offset of the camera device can be measured online, and the change of the posture of the camera device can be discovered in time.

[0096] The above-mentioned data acquisition module is used to form a positioning device by combining a camera device and a corresponding inertial measurement unit, and to set the positioning device in front of a calibration plate; to move the positioning device within a preset range through an external mechanical device, and to determine that the field of view of the camera device includes the calibration plate during the movement of the positioning device; and to obtain the posture data collected by each camera device and the inertial measurement unit corresponding to each camera device during the movement of the positioning device.

[0097] The data acquisition module is used to send a signal to each camera device and the corresponding inertial measurement unit through an external trigger; each camera device collects posture data based on the signal; and the inertial measurement unit corresponding to each camera device collects inertial data based on the signal.

[0098] The acquisition frequency of the camera device is lower than the acquisition frequency of the inertial measurement unit. The device further comprises: a data alignment module, which is used to align the posture data with the inertial data by interpolation.

[0099] The above-mentioned data alignment module is used to smooth the inertial data; wherein the smoothing process includes: Kalman filtering; determining the time delay value of the attitude data and the inertial data based on the time series of the attitude data and the time series of the inertial data; and aligning the attitude data with the smoothed inertial data by interpolation based on the time delay value.

[0100] The camera device correction module is used to determine that the posture of the camera device has changed if the change between the current inertial data collected by the inertial measurement unit and the historical inertial data collected at the previous moment is greater than a preset threshold; if it is determined that the posture of the camera device has changed, determine the posture change data of the camera device based on the change and the transformation matrix; determine the offset of the camera device based on the posture change data of the camera device, and correct the camera device based on the offset.

[0101] The camera device correction module is also used to obtain historical posture data of the camera device; and determine current posture data of the camera device based on the posture change data and the historical posture data.

[0102] The above-mentioned target includes a vehicle, and a plurality of camera devices locate the vehicle by collecting surface features of the vehicle.

[0103] The above-mentioned imaging device includes: a camera or a laser radar.

[0104] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the correction system of the camera device described above can refer to the corresponding process in the embodiment of the correction method of the camera device described above, and will not be repeated here.

[0105] Embodiment 4:

[0106] The embodiment of the present invention further provides an electronic device for executing the calibration method of the above-mentioned camera device; Figure 6 A structural schematic diagram of an electronic device is shown, which includes a memory 100 and a processor 101, wherein the memory 100 is used to store one or more computer instructions, and the one or more computer instructions are executed by the processor 101 to implement the correction method of the above-mentioned camera device.

[0107] Further, Figure 6 The electronic device shown further includes a bus 102 and a communication interface 103 , and the processor 101 , the communication interface 103 and the memory 100 are connected via the bus 102 .

[0108] The memory 100 may include a high-speed random access memory (RAM), and may also include a non-volatile memory, such as at least one disk storage. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 103 (which may be wired or wireless), and the Internet, wide area network, local area network, metropolitan area network, etc. may be used. The bus 102 may be an ISA bus, a PCI bus, or an EISA bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 6 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0109] The processor 101 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 101. The above processor 101 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present invention can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present invention can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 100, and the processor 101 reads the information in the memory 100 and completes the steps of the method of the above embodiment in combination with its hardware.

[0110] An embodiment of the present invention also provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the correction method of the above-mentioned camera device. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0111] The camera device correction method and the computer program product of the device provided in the embodiments of the present invention include a computer-readable storage medium storing program code. The instructions included in the program code can be used to execute the method in the previous method embodiment. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0112] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system and / or device described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0113] In addition, in the description of the embodiments of the present invention, unless otherwise clearly specified and limited, the terms "installed", "connected", and "connected" should be understood in a broad sense, for example, it can be a fixed connection, a detachable connection, or an integral connection; it can be a mechanical connection or an electrical connection; it can be a direct connection or an indirect connection through an intermediate medium, and it can be the internal communication of two components. For ordinary technicians in this field, the specific meanings of the above terms in the present invention can be understood according to specific circumstances.

[0114] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0115] In the description of the present invention, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer", etc., indicating the orientation or positional relationship, are based on the orientation or positional relationship shown in the drawings, and are only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as limiting the present invention. In addition, the terms "first", "second", and "third" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance.

[0116] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.

Claims

1. A calibration method for a camera device, characterized in that: A plurality of camera devices are used to locate a target, each of the camera devices is provided with a corresponding inertial measurement unit, and the method comprises: Acquire the posture data collected by each of the camera devices and the inertial data collected by the inertial measurement unit corresponding to each of the camera devices; the posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes acceleration data and angular velocity data of the inertial measurement unit; Determine a transformation matrix between a coordinate system of each of the inertial measurement units and a coordinate system of the corresponding camera device based on the posture data and the inertial data; Correcting the camera device based on current inertial data collected by the inertial measurement unit and the transformation matrix; The step of correcting the camera device based on the current inertial data collected by the inertial measurement unit and the transformation matrix includes: if the change between the current inertial data collected by the inertial measurement unit and the historical inertial data collected at the previous moment is greater than a preset threshold, determining that the posture of the camera device has changed; if it is determined that the posture of the camera device has changed, determining the posture change data of the camera device based on the change and the transformation matrix; determining the offset of the camera device based on the posture change data of the camera device, and correcting the camera device based on the offset.

2. The method according to claim 1, characterized in that The step of acquiring the posture data collected by each of the camera devices and the inertial data collected by the inertial measurement unit corresponding to each of the camera devices comprises: The camera device and the corresponding inertial measurement unit form a positioning device, and the positioning device is arranged in front of the calibration plate; The positioning device is moved within a preset range by an external mechanical device, and it is determined that the field of view of the camera device includes the calibration plate during the movement of the positioning device; During the movement of the positioning device, the posture data collected by each camera device and the inertial data collected by the inertial measurement unit corresponding to each camera device are obtained.

3. The method according to claim 2, characterized in that The step of acquiring the posture data collected by each of the camera devices and the inertial data collected by the inertial measurement unit corresponding to each of the camera devices comprises: Sending a signal to each of the camera devices and the corresponding inertial measurement unit via an external trigger; Each of the camera devices collects posture data based on the signal; The inertial measurement unit corresponding to each of the camera devices collects inertial data based on the signal.

4. The method according to claim 3, characterized in that The acquisition frequency of the camera device is less than the acquisition frequency of the inertial measurement unit, and the method further includes: The posture data is aligned with the inertial data by interpolation.

5. The method according to claim 4, characterized in that The step of aligning the posture data with the inertial data by interpolation comprises: Smoothing the inertial data; wherein the smoothing includes: Kalman filtering; Determining a time delay value between the posture data and the inertial data based on the time series of the posture data and the time series of the inertial data; The posture data is aligned with the smoothed inertial data by interpolation based on the time delay value.

6. The method according to claim 1, characterized in that After the step of determining the posture change data of the camera device based on the change amount and the transformation matrix, the method further includes: Acquiring historical posture data of the camera device; The current posture data of the camera device is determined based on the posture change data and the historical posture data.

7. The method according to any one of claims 1 to 6, characterized in that: The target includes a vehicle, and the plurality of camera devices locate the vehicle by capturing surface features of the vehicle.

8. The method according to any one of claims 1 to 6, characterized in that: The imaging device includes: a camera or a laser radar.

9. A calibration device for a camera device, characterized in that: A plurality of camera devices are used to locate a target, each of the camera devices is provided with a corresponding inertial measurement unit, and the device comprises: A data acquisition module, used to acquire the posture data collected by each of the camera devices and the inertial data collected by the inertial measurement unit corresponding to each of the camera devices; the posture data represents the posture of the camera device equivalent to the calibration plate, and the inertial data includes acceleration data and angular velocity data of the inertial measurement unit; A transformation matrix determination module, used for determining a transformation matrix between a coordinate system of each of the inertial measurement units and a coordinate system of the corresponding camera device based on the posture data and the inertial data; A camera device correction module, used to correct the camera device based on the current inertial data collected by the inertial measurement unit and the transformation matrix; The camera device correction module is used to determine that the posture of the camera device has changed if the change between the current inertial data collected by the inertial measurement unit and the historical inertial data collected at the previous moment is greater than a preset threshold; if it is determined that the posture of the camera device has changed, determine the posture change data of the camera device based on the change and the transformation matrix; determine the offset of the camera device based on the posture change data of the camera device, and correct the camera device based on the offset.

Images