Camera calibration method and device, electronic equipment, storage medium and program product
Camera calibration is performed by collecting images by mobile robots and chassis rotation, which solves the problems of low calibration efficiency and high maintenance costs in the prior art, and realizes an automated and simplified operational camera calibration process.
Patent Information
- Application Number
- CN202411302118.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-09-18
- Publication Date
- 2025-01-07
AI Technical Summary
The existing camera calibration methods require specific technicians to complete, and the operation is complicated, resulting in low calibration efficiency and high maintenance costs of robot products.
By controlling the movement of the mobile robot and the rotation of the chassis, the camera itself collects calibrator images of different distances and angles, and calibrates the pitch angle and position based on the image set, and uses weighted aggregation calculation and abnormal data removal to realize automatic calibration of the camera.
Camera calibration can be completed without the need for technicians to collect additional tool parts, simplifying operations, improving calibration efficiency, and reducing maintenance costs of robot products.
Smart Images
Figure CN119273771A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robotics, and in particular to a camera calibration method, device, electronic device, storage medium and program product. Background Art
[0002] With the rapid development of computer vision technology and the improvement of hardware computing speed, mobile robots have been rapidly developed and widely used. Depth camera (i.e. 3D addition) is an important way for robots to obtain external information. Depth camera can collect color images and measure the distance from the spatial object to the camera. Compared with laser sensors, depth cameras are low-cost, rich in information, and flexible in installation. Therefore, they are often used as visual sensors for mobile robots. In order to control the motion of mobile robots, it is necessary to transform the camera pose estimated by the visual sensor signal into the robot pose. Solving this pose transformation is the calibration problem of the camera relative to the robot chassis.
[0003] The depth camera installed on the mobile robot has been calibrated before leaving the factory. However, when the user uses the mobile robot, the position of the depth camera may change due to bumps or bumps. In this case, the depth camera needs to be recalibrated.
[0004] In related technologies, calibration work needs to be completed by experienced technicians. Technicians can use additional acquisition tooling to obtain the corresponding images and laser point cloud data of the robot at different positions, and use complex alignment algorithms to solve the position deviation of the depth camera, thereby calibrating the depth camera.
[0005] However, existing camera calibration methods require specific technicians to complete and are complex to operate, resulting in low camera calibration efficiency and high maintenance costs for robot products. Summary of the invention
[0006] The embodiments of the present application provide a camera calibration method, device, electronic device, storage medium and program product to solve the technical problems of low calibration efficiency of existing camera calibration methods and high maintenance cost of robot products.
[0007] In a first aspect, an embodiment of the present application provides a camera calibration method, which is applied to a mobile robot, wherein the mobile robot includes a chassis and a fixedly installed camera, and the method includes: controlling the mobile robot to move, and photographing a calibration object through the camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; based on the first image set, calibrating the pitch angle of the camera, and determining the pitch angle calibration result of the camera; controlling the mobile robot to move to the front of the calibration object; controlling the chassis to rotate, and photographing the calibration object through the camera to obtain a second image set; the second image set includes images of the calibration object taken at different rotation angles; based on the second image set and the rotation angle information of the chassis, calibrating the relative position of the camera and the chassis, and determining the position calibration result of the camera.
[0008] In one embodiment, the rotation angle information includes the rotation angle of the chassis corresponding to each calibration object image in the second image set; based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera, including: obtaining the camera intrinsic parameter data and calibration object data of the camera; based on the camera intrinsic parameter data, the calibration object data, each calibration object image in the second image set and the rotation angle of the chassis corresponding to each calibration object image in the second image set, the relative position of the camera and the chassis is calibrated to determine the position deviation value; multiple position deviation values are obtained, and based on the multiple position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera.
[0009] In one embodiment, a method for determining a position deviation value includes: selecting any two calibration object images in a second image set, respectively serving as a first calibration object image and a second calibration object image; determining a first position vector and a second position vector respectively based on camera intrinsic parameter data and calibration object data; the first position vector is a position vector between the calibration object and the camera in the first calibration object image, and the second position vector is a position vector between the calibration object and the camera in the second calibration object image; determining a distance between the first position and the second position based on the first position vector and the second position vector; the first position is a camera position of the camera when shooting the first calibration object image, and the second position is a camera position of the camera when shooting the second calibration object image; based on a plane geometric relationship between the first position, the second position and a center point of the chassis, a rotation angle corresponding to the first calibration object image and a rotation angle corresponding to the second calibration object image, calibrating the relative position of the camera and the chassis, and calculating a position deviation value.
[0010] In one embodiment, a weighted aggregation calculation is performed based on multiple position deviation values to obtain a position calibration result of the camera, including: selecting a position deviation value as reference data and using the remaining position deviation values as comparison data; respectively determining the distance difference between each comparison data and the reference data; respectively comparing each distance difference with a first preset threshold to obtain multiple comparison results; based on each comparison result, determining the error count of the reference data; returning to the step of selecting a position deviation value as reference data and using the remaining position deviation values as comparison data until the error counts corresponding to all position deviation values are obtained; based on the error count corresponding to each position deviation value, abnormal data is eliminated to obtain multiple normal position deviation values; based on the multiple normal position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera.
[0011] In one embodiment, based on the first image set, the pitch angle of the camera is calibrated to determine the pitch angle calibration result of the camera, including: obtaining camera extrinsic parameter data of the camera, camera intrinsic parameter data of the camera and calibration object data; based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data and the first image set, the pitch angle of the camera is calibrated to determine multiple pitch angles; based on the multiple pitch angles, a weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.
[0012] In one embodiment, a weighted aggregation calculation is performed based on multiple pitch angles to obtain a pitch angle calibration result of the camera, including: selecting a pitch angle as reference data and using the remaining pitch angles as comparison data; respectively determining the angle difference between each comparison data and the reference data; respectively comparing each angle difference with a second preset threshold to obtain multiple comparison results; based on each comparison result, determining an error count of the reference data; returning to the step of selecting a pitch angle as reference data and using the remaining pitch angles as comparison data until the error counts corresponding to all pitch angles are obtained; based on the error count corresponding to each pitch angle, abnormal data is eliminated to obtain multiple normal pitch angles; based on the multiple normal pitch angles, weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.
[0013] In a second aspect, an embodiment of the present application provides a camera calibration device, comprising: a shooting module, used to control the movement of a mobile robot, and to shoot a calibration object through a camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; a pitch angle calibration module, used to calibrate the pitch angle of the camera based on the first image set, and determine the pitch angle calibration result of the camera; a control module, used to control the mobile robot to move to the front of the calibration object; the shooting module, also used to control the rotation of the chassis, and to shoot the calibration object through the camera to obtain a second image set; the second image set includes images of the calibration object shot at different rotation angles; a position calibration module, used to calibrate the relative position of the camera and the chassis based on the second image set and the rotation angle information of the chassis, and determine the position calibration result of the camera.
[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned camera calibration methods is implemented.
[0015] In a fourth aspect, an embodiment of the present application provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-mentioned camera calibration methods.
[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any of the above-mentioned camera calibration methods when executed by a processor.
[0017] The camera calibration method, device, electronic device, storage medium and program product provided in the embodiment of the present application control the movement of the mobile robot, and photograph the calibration object through the camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; based on the first image set, the pitch angle of the camera is calibrated to determine the pitch angle calibration result of the camera; the mobile robot is controlled to move to the front of the calibration object; the chassis is controlled to rotate, and the calibration object is photographed through the camera to obtain a second image set; the second image set includes images of the calibration object taken at different rotation angles; based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera. Through the above method, there is no need for the technician to use additional collection tooling to perform data collection and camera calibration. The camera calibration can be completed by using the images collected by the camera set by the robot itself. The operation is simple, the efficiency of camera calibration can be improved, and the maintenance cost of the robot product can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS
[0018] In order to more clearly illustrate the technical solutions in the present application or the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.
[0019] Figure 1 It is a flowchart of the camera calibration method provided in an embodiment of the present application.
[0020] Figure 2 It is a schematic diagram of capturing a first image provided in an embodiment of the present application.
[0021] Figure 3 It is a three-dimensional view of the position calibration process provided by an embodiment of the present application.
[0022] Figure 4 This is a top view of the robot during the position calibration process provided in an embodiment of the present application.
[0023] Figure 5 It is a structural schematic diagram of a camera calibration device provided in an embodiment of the present application.
[0024] Figure 6 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solutions and advantages of this application clearer, the technical solutions in this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0026] See also Figure 1 , Figure 1 : is a flowchart of a camera calibration method provided in an embodiment of the present application. In an embodiment of the present application, the camera calibration method is applied to a mobile robot, the mobile robot includes a chassis and a fixedly arranged camera, and the camera calibration method includes steps S110 to S150, each of which is as follows: S110: Control the mobile robot to move, and use a camera to photograph the calibration object to obtain a first image set.
[0027] The first image set includes images of the calibration object at different shooting distances and / or different shooting angles.
[0028] Specifically, the calibration object is first fixed at a certain position (such as on a wall) so that the center of the calibration object is substantially at the same height as the camera fixed on the mobile robot, and ensure that there is no debris in the area around the calibration object to avoid interference with the movement of the mobile robot.
[0029] Optionally, the calibration object is a calibration plate.
[0030] See also Figure 2 , Figure 2 It is a schematic diagram of capturing a first image provided in an embodiment of the present application.
[0031] like Figure 2 As shown, fix the calibration plate on the wall. The center of the calibration plate is roughly consistent with the height of the camera fixed on the mobile robot. Make sure there are no debris in the surrounding area to avoid interference with the movement of the robot.
[0032] Furthermore, the mobile robot is controlled to move, and the calibration object is photographed by a camera to obtain a first image set.
[0033] by Figure 2 For example, the mobile robot can be controlled to move to a position about 2 meters away from the calibration plate, and the pitch angle calibration program can be started. The mobile robot will move autonomously according to the set pitch angle calibration program, and use the camera to shoot the calibration plate to collect a series of calibration object images at different shooting distances and / or different shooting angles to form a first image set.
[0034] S120: Calibrate the pitch angle of the camera based on the first image set to determine a calibration result of the pitch angle of the camera.
[0035] Specifically, after obtaining the first image set, the pitch angle calibration program can process and analyze the collected first image set, and use the PNP algorithm to solve the camera extrinsic parameter data, combine the camera intrinsic parameter data and the calibration object data (that is, the known parameters of the calibration object) to calibrate the camera's pitch angle, and calculate the camera's attitude angle relative to the calibration object. The attitude angle is recorded as (Roll, Pitch, Yaw), where Roll represents the roll angle, Yaw represents the yaw angle, and Pitch represents the pitch angle. The calculated Pitch angle is the pitch angle of the 3D camera relative to the moving plane of the mobile robot. By calculating multiple sets of Pitch angle data and performing weighted aggregation calculations, the final pitch angle calibration result of the camera can be determined.
[0036] S130: Control the mobile robot to move to the front of the calibration object.
[0037] Specifically, the mobile robot may be controlled to move to a position about 0.5 meters in front of the calibration object so that the mobile robot faces the calibration object.
[0038] S140: Control the chassis to rotate, and use a camera to photograph the calibration object to obtain a second image set.
[0039] The second image set includes images of the calibration object taken at different rotation angles.
[0040] When the mobile robot is facing the calibration object, the position calibration program is started. The mobile robot will automatically control the chassis rotation according to the set position calibration program, and shoot the calibration object through the camera to obtain the second image set.
[0041] Specifically, the mobile robot can first collect a reference image I according to a set position calibration program, collect the current calibration object image by controlling the chassis to rotate at a small angle θ, and record the current rotation angle θ; the calibration object is photographed by rotating at different angles, and the calibration object images photographed at different rotation angles are collected to form a second image set, generating multiple groups of data, each group of data including a calibration object image and the rotation angle corresponding to the calibration object image (i.e., the rotation angle of the chassis).
[0042] S150: Based on the second image set and the rotation angle information of the chassis, calibrate the relative position of the camera and the chassis to determine the camera position calibration result.
[0043] Specifically, based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated, and the horizontal position offset of the 3D camera relative to the center of the mobile robot chassis is calculated, that is, the position calibration result of the camera.
[0044] The camera calibration method provided in the embodiment of the present application controls the movement of the mobile robot, and photographs the calibration object through the camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; based on the first image set, the pitch angle of the camera is calibrated to determine the pitch angle calibration result of the camera; the mobile robot is controlled to move to the front of the calibration object; the chassis is controlled to rotate, and the calibration object is photographed through the camera to obtain a second image set; the second image set includes images of the calibration object taken at different rotation angles; based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera. Through the above method, there is no need for the technician to use additional collection tooling to perform data collection and camera calibration. The camera calibration can be completed by using the images collected by the camera set by the robot itself. The operation is simple, the efficiency of camera calibration can be improved, and the maintenance cost of the robot product can be reduced.
[0045] In some embodiments, the rotation angle information includes the rotation angle of the chassis corresponding to each calibration object image in the second image set; based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera, including: obtaining the camera intrinsic parameter data and calibration object data of the camera; based on the camera intrinsic parameter data, the calibration object data, each calibration object image in the second image set and the rotation angle of the chassis corresponding to each calibration object image in the second image set, the relative position of the camera and the chassis is calibrated to determine the position deviation value; multiple position deviation values are obtained, and based on the multiple position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera.
[0046] Specifically, the camera intrinsic parameter data and calibration object data of the camera are obtained.
[0047] The calibration object data refers to the known parameters of the calibration object.
[0048] In computer vision, especially in the field of camera calibration and stereo vision, camera intrinsic parameters and camera extrinsic parameters are very important concepts, both of which are related to the geometric properties and posture of the camera.
[0049] Among them, the camera intrinsic data is the parameters that describe the internal properties of the camera, including focal length, principal point (optical center) coordinates, distortion coefficients, etc. The camera intrinsic data is usually determined during camera calibration. The camera intrinsic data of a specific camera model is usually fixed and does not change over time. Once the camera intrinsic data is determined, it usually remains unchanged during the use of the camera.
[0050] Camera extrinsic data are parameters that describe the position and attitude of the camera in the world coordinate system, usually including rotation matrix and translation vector. Camera extrinsic data may change at different camera positions or shooting times. For example, in stereo vision, assuming there are two cameras, the relative position and direction between the two cameras will change every time the camera is moved, resulting in changes in the camera extrinsic data.
[0051] Furthermore, based on the camera intrinsic parameter data, the calibration object data, each calibration object image in the second image set and the rotation angle of the chassis corresponding to each calibration object image in the second image set, the relative position of the camera and the chassis is calibrated to determine the position deviation value.
[0052] Furthermore, a plurality of position deviation values are obtained, and based on the plurality of position deviation values, a weighted aggregation calculation is performed to obtain a position calibration result of the camera.
[0053] In some embodiments, a method for determining a position deviation value includes: selecting any two calibration object images in a second image set, respectively as a first calibration object image and a second calibration object image; determining a first position vector and a second position vector respectively based on camera intrinsic parameter data and calibration object data; the first position vector is a position vector between the calibration object and the camera in the first calibration object image, and the second position vector is a position vector between the calibration object and the camera in the second calibration object image; determining a distance between the first position and the second position based on the first position vector and the second position vector; the first position is a camera position of the camera when shooting the first calibration object image, and the second position is a camera position of the camera when shooting the second calibration object image; based on the plane geometric relationship between the first position, the second position and the center point of the chassis, the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image, calibrating the relative position of the camera and the chassis, and calculating a position deviation value.
[0054] After obtaining the camera's intrinsic parameter data and calibration object data, the camera position can be calibrated.
[0055] See also Figure 3 and Figure 4 , Figure 3 is a three-dimensional view of the position calibration process provided by the embodiment of the present application, Figure 4 FIG. 1 is a top view of a robot during the position calibration process provided in an embodiment of the present application. Figure 3 and Figure 4 As shown, in order to simplify the structure of the mobile robot, Figure 3 In the figure, a cylinder is used to represent the mobile robot, and a black and white rectangle is used to represent the calibration object. Figure 4 The circle in Figure 3 Top view of the mobile robot in .
[0056] like Figure 3 As shown, in this embodiment, the camera is fixedly mounted on the mobile robot. Therefore, when the mobile robot performs a self-rotation action (i.e., controls the chassis to rotate), the camera will make a circular motion with the center point o of the mobile robot chassis as the center and the position deviation length length as the radius. Specifically, after the mobile robot rotates an angle, the camera position before and after the rotation will form an isosceles triangle with the center point o.
[0057] Specifically, after obtaining the camera's intrinsic parameter data and calibration object data, the three-axis translation vector of the camera relative to the calibration plate can be calculated through the OpenCV hand-eye calibration function. .
[0058] Further, any two calibration object images are selected in the second image set as the first calibration object image i and the second calibration object image j, respectively, and the first position vector is determined based on the camera intrinsic parameter data and the calibration object data. and the second position vector .
[0059] First position vector is the position vector between the calibration object and the camera in the first calibration object image, and the second position vector is the position vector between the calibration object and the camera in the second calibration object image.
[0060] Furthermore, it can be seen from the spatial vector relationship that based on the first position vector and the second position vector , the first position c can be determined i With the second position c j The distance between ,distance The calculation formula is as follows: ; The first position is the camera position when the camera captures the first calibration object image, and the second position is the camera position when the camera captures the second calibration object image.
[0061] Further, based on the first position c i , second position c j The relative position of the camera and the chassis is calibrated based on the plane geometric relationship with the center point o of the chassis, the rotation angle corresponding to the first calibration object image, and the rotation angle corresponding to the second calibration object image, and a position deviation value is calculated.
[0062] For ease of understanding, Figure 3 Let's take this as an example to explain.
[0063] like Figure 3 As shown, any two calibration object images are selected in the second image set, which are respectively recorded as the first calibration object image 1 and the second calibration object image 2, and the first position vector is determined based on the camera intrinsic parameter data and the calibration object data. and the second position vector ; From the spatial vector relationship, we can see that based on the first position vector and the second position vector , the distance between the first position c1 and the second position c2 can be determined .
[0064] Assuming that the mobile robot performs a self-rotation action (i.e., controls the rotation of the chassis), the camera moves in a circle with the center point o of the mobile robot chassis as the center and the position deviation length as the radius. Then the first position c1, the second position c2 and the center point o of the chassis can form an isosceles triangle with a vertex angle of θ. The vertex angle θ is the rotation angle of the chassis, which can be determined based on the self-rotation angle corresponding to the first calibration object image and the self-rotation angle corresponding to the second calibration object image.
[0065] like Figure 4 As shown, at this time, based on the first position c i , second position c j The plane geometric relationship with the center point o of the chassis can be used to obtain the base angle α of the isosceles triangle. The calculation formula for the base angle α is as follows: .
[0066] According to the plane geometric relationship, the relative position of the camera and the chassis can be calibrated and a position deviation value length can be calculated. The calculation formula of the position deviation value length is as follows: .
[0067] Furthermore, the process returns to the step of selecting any two calibration object images in the second image set until a plurality of position deviation values are calculated.
[0068] Specifically, based on the collected multiple calibration object images in the second image set, multiple position deviation values are calculated and respectively recorded as length1, length2, length3, length4, etc.
[0069] Furthermore, based on multiple position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera .
[0070] The camera calibration method provided in the embodiment of the present application collects calibration object image data of different perspectives through the rotation of the mobile robot, combines the angle data of the chassis rotation, and utilizes the geometric relationship between space and plane. Through simple calculation, the relative position of the camera and the chassis can be calibrated, and the accurate calculation of the position deviation of the camera relative to the chassis of the mobile robot can be achieved.
[0071] In some embodiments, a weighted aggregation calculation is performed based on multiple position deviation values to obtain a position calibration result of the camera, including: selecting a position deviation value as reference data and using the remaining position deviation values as comparison data; respectively determining the distance difference between each comparison data and the reference data; respectively comparing each distance difference with a first preset threshold to obtain multiple comparison results; based on each comparison result, determining the error count of the reference data; returning to the step of selecting a position deviation value as reference data and using the remaining position deviation values as comparison data until the error counts corresponding to all position deviation values are obtained; based on the error count corresponding to each position deviation value, abnormal data is eliminated to obtain multiple normal position deviation values; based on the multiple normal position deviation values, weighted aggregation calculation is performed to obtain the position calibration result of the camera.
[0072] It is understandable that when collecting the calibration object image, when the robot moves slightly or the shooting angle is poor, the collected calibration object image may contain blurred or unclear images, and such image data will affect the calculation of the position deviation value.
[0073] Therefore, it is necessary to eliminate abnormal position deviation values and then perform weighted aggregation calculation to effectively improve the accuracy of the position calibration results.
[0074] In this embodiment, a statistical voting method is used to identify and eliminate abnormal position deviation values. The specific steps are as follows: (1) Select one position deviation value as the reference data, and use the remaining position deviation values as the comparison data.
[0075] Specifically, among the multiple position deviation values, the first position deviation value is selected as reference data, and the remaining position deviation values are used as comparison data.
[0076] (2) Determine the distance difference between each comparison data and the reference data respectively.
[0077] Specifically, each remaining position deviation value is sequentially compared with the reference data (ie, the first position deviation value) for analysis: for each remaining position deviation value, the distance difference between the remaining position deviation value and the reference data is calculated.
[0078] For example, calculate the second position deviation value and reference data The distance difference between , the calculation formula is as follows: .
[0079] (3) Each distance difference The first and second preset thresholds are compared respectively to obtain a plurality of comparison results.
[0080] Optionally, the first preset threshold is 0.1 m (meter).
[0081] For each distance difference , if the distance difference is greater than the first preset threshold, it can be considered that there is a large error between the remaining position deviation value and the reference data; if the distance difference If the error between the remaining position deviation value and the reference data is smaller than or equal to the first preset threshold, it can be considered that the error between the remaining position deviation value and the reference data is small.
[0082] (4) Based on each comparison result, determine the error count of the reference data.
[0083] Specifically, if the distance difference If the error value is greater than the first preset threshold, the error count of the reference data is increased by 1. The error count of the reference data can be determined based on each comparison result.
[0084] It is understandable that the error count of a reference data should start increasing from 0.
[0085] (5) Return to the step of selecting a position deviation value as reference data and using the remaining position deviation values as comparison data until the error counts corresponding to all position deviation values are obtained.
[0086] Specifically, for each reference data, loop through all other position deviation values, complete a round of comparison, and then replace the new reference data, and use length2, length3, length4, etc. as reference data in turn, and repeat the above comparison process until the error counts corresponding to all position deviation values are obtained.
[0087] (6) Based on the error count corresponding to each position deviation value, abnormal data is eliminated to obtain multiple normal position deviation values; based on the multiple normal position deviation values, weighted aggregation calculation is performed to obtain the camera position calibration result.
[0088] Specifically, based on each position deviation value, the error count values are sorted from large to small to obtain a sorting result; the top 20% of the position deviation values in the sorting result are taken as abnormal data, the top 20% of the position deviation values in the sorting result are eliminated, and the remaining position deviation values are normal position deviation values.
[0089] Furthermore, based on multiple normal position deviation values, a weighted aggregation calculation is performed to obtain a position calibration result of the camera.
[0090] Considering that the calculation of abnormal position deviation values is based on images, when abnormal position deviation values appear, there may be problems with the image. Therefore, the image corresponding to the abnormal position deviation value should be eliminated, and then the camera position calibration result should be recalculated based on the retained normal image.
[0091] Preferably, based on each position deviation value, the error count values are sorted from large to small. After the sorting results are obtained, the calibration object images corresponding to the top 20% position deviation values in the sorting results are taken as abnormal images, all abnormal images are eliminated, and multiple normal images are retained; based on the normal images, the relative position of the camera and the chassis is calibrated to determine the camera position calibration result.
[0092] It should be noted that in the prior art, there are few studies on the accuracy verification of data sets. Most studies assume that the collected data sets are normal and can be used directly, so the collected image data are not verified and screened. The present application performs a secondary screening of the collected images, traverses the position deviation values, compares the differences between them, eliminates image data based on statistical voting results, and calculates the final position calibration results based on the remaining images, which can further improve the accuracy of the calibration results.
[0093] The camera calibration method provided in the embodiment of the present application adopts a data elimination strategy to eliminate abnormal data, identifies and eliminates abnormal data by means of statistical voting, thereby ensuring the reliability and accuracy of the final result and enhancing the robustness of the algorithm.
[0094] In some embodiments, based on the first image set, the pitch angle of the camera is calibrated to determine the pitch angle calibration result of the camera, including: obtaining the camera extrinsic parameter data, the camera intrinsic parameter data and the calibration object data of the camera; based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data and the first image set, the pitch angle of the camera is calibrated to determine multiple pitch angles; based on the multiple pitch angles, a weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.
[0095] Specifically, after obtaining the first image set, the pitch angle calibration program can process and analyze the collected first image set, solve the camera extrinsic parameter data of the camera by using the PNP algorithm, and obtain the camera intrinsic parameter data and calibration object data (i.e., the known parameters of the calibration object) of the camera.
[0096] Furthermore, based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data and the first image set, the pitch angle of the camera is calibrated, and multiple attitude angles of the camera relative to the calibration object are calculated. The attitude angles are recorded as (Roll, Pitch, Yaw), where Roll represents the roll angle, Yaw represents the yaw angle, and Pitch represents the pitch angle.
[0097] Furthermore, based on multiple pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.
[0098] The camera calibration method provided in the embodiment of the present application realizes the camera pitch angle calibration by collecting multiple groups of calibration object images and combining the extrinsic parameter calibration algorithm and the weighted average method. The operation is simple and the calculation complexity is low.
[0099] In some embodiments, a weighted aggregation calculation is performed based on multiple pitch angles to obtain a pitch angle calibration result of the camera, including: selecting a pitch angle as reference data and using the remaining pitch angles as comparison data; respectively determining the angle difference between each comparison data and the reference data; respectively comparing each angle difference with a second preset threshold to obtain multiple comparison results; based on each comparison result, determining an error count of the reference data; returning to the step of selecting a pitch angle as reference data and using the remaining pitch angles as comparison data until the error counts corresponding to all pitch angles are obtained; based on the error count corresponding to each pitch angle, abnormal data is eliminated to obtain multiple normal pitch angles; based on the multiple normal pitch angles, a weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.
[0100] It is understandable that when collecting the calibration object image, if the robot moves slightly or the shooting angle is poor, the collected calibration object image may contain blurred or unclear images, and such image data will affect the calculation of the pitch angle.
[0101] Therefore, it is necessary to eliminate abnormal pitch angles and then perform weighted aggregation calculation to effectively improve the accuracy of the pitch angle calibration results.
[0102] In this embodiment, a statistical voting method is used to identify and eliminate abnormal pitch angles. The specific steps are as follows: (1) Select one pitch angle as the reference data and the remaining pitch angles as the comparison data.
[0103] Specifically, among the multiple pitch angles, the first pitch angle is selected as reference data, and the remaining pitch angles are used as comparison data.
[0104] (2) Determine the angle difference between each comparison data and the reference data.
[0105] Specifically, each remaining pitch angle is sequentially compared and analyzed with the reference data (ie, the first pitch angle): for each remaining pitch angle, the angle difference between the remaining pitch angle and the reference data is calculated.
[0106] (3) Compare each angle difference with the second preset threshold value to obtain multiple comparison results.
[0107] For each angle difference, if the angle difference is greater than the second preset threshold, it can be considered that there is a large error between the remaining pitch angles and the reference data; if the angle difference is less than or equal to the second preset threshold, it can be considered that there is a small error between the remaining pitch angles and the reference data.
[0108] (4) Based on each comparison result, determine the error count of the reference data.
[0109] Specifically, if the angle difference is greater than the second preset threshold, the error count of the reference data is increased by 1, and the error count of the reference data can be determined based on each comparison result.
[0110] It is understandable that the error count of a reference data should start increasing from 0.
[0111] (5) Return to the step of selecting one pitch angle as reference data and the remaining pitch angles as comparison data until the error counts corresponding to all pitch angles are obtained.
[0112] (6) Based on the error count corresponding to each pitch angle, abnormal data is eliminated to obtain multiple normal pitch angles; based on the multiple normal pitch angles, weighted aggregation calculation is performed to obtain the pitch angle calibration result of the camera.
[0113] Specifically, based on each pitch angle, the error count values are sorted from large to small to obtain a sorting result; the first 20% of the pitch angles in the sorting result are taken as abnormal data, the first 20% of the pitch angles in the sorting result are eliminated, and the remaining pitch angles are normal pitch angles.
[0114] Furthermore, based on multiple normal pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.
[0115] Considering that the calculation of abnormal pitch angles is based on images, when a pitch angle appears, there may be a problem with the image. Therefore, the image corresponding to the abnormal pitch angle should be eliminated, and then the camera pitch angle calibration result should be recalculated based on the retained normal image.
[0116] Preferably, based on each pitch angle, the error count values are sorted from large to small. After the sorting results are obtained, the calibration object images corresponding to the first 20% of the pitch angles in the sorting results are taken as abnormal images, all abnormal images are eliminated, and multiple normal images are retained; based on the normal images, the pitch angle of the camera is calibrated to determine the pitch angle calibration result of the camera.
[0117] The present application also provides a camera calibration device. Figure 5 , Figure 55 is a schematic diagram of the structure of a camera calibration device provided in an embodiment of the present application. In the embodiment of the present application, the camera calibration device includes a shooting module 510, a pitch angle calibration module 520, a control module 530 and a position calibration module 540.
[0118] The shooting module 510 is used to control the movement of the mobile robot and shoot the calibration object through a camera to obtain a first image set.
[0119] The first image set includes images of the calibration object at different shooting distances and / or different shooting angles.
[0120] The pitch angle calibration module 520 is used to calibrate the pitch angle of the camera based on the first image set, and determine the pitch angle calibration result of the camera.
[0121] The control module 530 is used to control the mobile robot to move to the front of the calibration object.
[0122] The shooting module 510 is also used to control the chassis rotation and shoot the calibration object through the camera to obtain a second image set.
[0123] The second image set includes images of the calibration object taken at different rotation angles.
[0124] The position calibration module 540 is used to calibrate the relative position of the camera and the chassis based on the second image set and the rotation angle information of the chassis, and determine the position calibration result of the camera.
[0125] In some embodiments, the rotation angle information includes the rotation angle of the chassis corresponding to each calibration object image in the second image set.
[0126] The position calibration module 540 is used to obtain the camera intrinsic parameter data and calibration object data of the camera; based on the camera intrinsic parameter data, the calibration object data, each calibration object image in the second image set and the rotation angle of the chassis corresponding to each calibration object image in the second image set, calibrate the relative position of the camera and the chassis to determine multiple position deviation values; based on the multiple position deviation values, perform weighted aggregation calculation to obtain the camera position calibration result.
[0127] In some embodiments, the position calibration module 540 is used to determine the position deviation value, including: selecting any two calibration object images in the second image set, respectively as the first calibration object image and the second calibration object image; based on the camera intrinsic parameter data and the calibration object data, respectively determining the first position vector and the second position vector; the first position vector is the position vector between the calibration object and the camera in the first calibration object image, and the second position vector is the position vector between the calibration object and the camera in the second calibration object image; based on the first position vector and the second position vector, determining the distance between the first position and the second position; the first position is the camera position of the camera when shooting the first calibration object image, and the second position is the camera position of the camera when shooting the second calibration object image; based on the plane geometric relationship between the first position, the second position and the center point of the chassis, the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image, calibrating the relative position of the camera and the chassis, and calculating a position deviation value.
[0128] In some embodiments, the position calibration module 540 is used to select a position deviation value as reference data and the remaining position deviation values as comparison data; respectively determine the distance difference between each comparison data and the reference data; respectively compare each distance difference with a first preset threshold value to obtain multiple comparison results; based on each comparison result, determine the error count of the reference data; return to the step of selecting a position deviation value as reference data and the remaining position deviation values as comparison data until the error counts corresponding to all position deviation values are obtained; based on the error count corresponding to each position deviation value, perform abnormal data elimination to obtain multiple normal position deviation values; based on the multiple normal position deviation values, perform weighted aggregation calculation to obtain the position calibration result of the camera.
[0129] In one embodiment, the pitch angle calibration module 520 is used to obtain camera extrinsic parameter data, camera intrinsic parameter data and calibration object data of the camera; based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data and the first image set, calibrate the pitch angle of the camera to determine multiple pitch angles; based on the multiple pitch angles, perform weighted aggregation calculation to obtain the pitch angle calibration result of the camera.
[0130] In one embodiment, the pitch angle calibration module 520 is used to select a pitch angle as reference data and use the remaining pitch angles as comparison data; respectively determine the angle difference between each comparison data and the reference data; respectively compare each angle difference with a second preset threshold value to obtain multiple comparison results; based on each comparison result, determine the error count of the reference data; return to the step of selecting a pitch angle as reference data and using the remaining pitch angles as comparison data until the error counts corresponding to all pitch angles are obtained; based on the error count corresponding to each pitch angle, perform abnormal data elimination to obtain multiple normal pitch angles; based on the multiple normal pitch angles, perform weighted aggregation calculation to obtain the pitch angle calibration result of the camera.
[0131] The present application also provides an electronic device, Figure 6 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application, such as Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the camera calibration method.
[0132] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.
[0133] An embodiment of the present application also provides a non-transitory computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the camera calibration method provided by the above methods is implemented.
[0134] An embodiment of the present application also provides a computer program product, which includes a computer program. The computer program can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the camera calibration methods provided by the above methods.
[0135] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.
[0136] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.
[0137] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.
Claims
1. A camera calibration method, characterized in that: Applied to a mobile robot, the mobile robot comprising a chassis and a fixedly arranged camera, the method comprising: Controlling the mobile robot to move, and photographing the calibration object through the camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; Calibrate the pitch angle of the camera based on the first image set, and determine a calibration result of the pitch angle of the camera; Controlling the mobile robot to move to the front of the calibration object; Controlling the chassis to rotate, and photographing the calibration object through the camera to obtain a second image set; the second image set includes images of the calibration object photographed at different rotation angles; Based on the second image set and the rotation angle information of the chassis, the relative position of the camera and the chassis is calibrated to determine the position calibration result of the camera.
2. The camera calibration method according to claim 1, characterized in that: The rotation angle information includes the rotation angle of the chassis corresponding to each calibration object image in the second image set; The step of calibrating the relative position of the camera and the chassis based on the second image set and the rotation angle information of the chassis, and determining the position calibration result of the camera includes: Acquiring camera intrinsic parameter data and calibration object data of the camera; Based on the camera intrinsic parameter data, the calibration object data, each calibration object image in the second image set, and the rotation angle of the chassis corresponding to each calibration object image in the second image set, calibrate the relative position of the camera and the chassis to determine the position deviation value; A plurality of the position deviation values are obtained, and based on the plurality of the position deviation values, a weighted aggregation calculation is performed to obtain a position calibration result of the camera.
3. The camera calibration method according to claim 2, characterized in that: The method for determining the position deviation value comprises: Selecting any two calibration object images from the second image set as the first calibration object image and the second calibration object image respectively; Based on the camera intrinsic parameter data and the calibration object data, a first position vector and a second position vector are determined respectively; the first position vector is a position vector between the calibration object and the camera in the first calibration object image, and the second position vector is a position vector between the calibration object and the camera in the second calibration object image; Determine a distance between a first position and a second position based on the first position vector and the second position vector; the first position is a camera position of the camera when taking the first calibration object image, and the second position is a camera position of the camera when taking the second calibration object image; Based on the planar geometric relationship between the first position, the second position and the center point of the chassis, the rotation angle corresponding to the first calibration object image and the rotation angle corresponding to the second calibration object image, the relative position of the camera and the chassis is calibrated to calculate the position deviation value.
4. The camera calibration method according to claim 2, characterized in that: The step of performing weighted aggregation calculation based on the plurality of position deviation values to obtain the position calibration result of the camera includes: Selecting one of the position deviation values as reference data, and using the remaining position deviation values as comparison data; respectively determining the distance difference between each of the comparison data and the reference data; Comparing each of the distance differences with a first preset threshold value to obtain a plurality of comparison results; determining an error count of the reference data based on each of the comparison results; Returning to the step of selecting one of the position deviation values as reference data and using the remaining position deviation values as comparison data, until error counts corresponding to all the position deviation values are obtained; Based on the error count corresponding to each position deviation value, abnormal data is eliminated to obtain a plurality of normal position deviation values; Based on the multiple normal position deviation values, a weighted aggregation calculation is performed to obtain the position calibration result of the camera.
5. The camera calibration method according to claim 1, characterized in that: The step of calibrating the pitch angle of the camera based on the first image set and determining the pitch angle calibration result of the camera includes: Acquire camera extrinsic parameter data of the camera, camera intrinsic parameter data of the camera, and calibration object data; Calibrate the pitch angle of the camera based on the camera extrinsic parameter data, the camera intrinsic parameter data, the calibration object data and the first image set to determine a plurality of pitch angles; Based on the multiple pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.
6. The camera calibration method according to claim 5, characterized in that: The step of performing weighted aggregation calculation based on the plurality of pitch angles to obtain a pitch angle calibration result of the camera includes: Selecting one of the pitch angles as reference data, and using the remaining pitch angles as comparison data; respectively determining the angle difference between each of the comparison data and the reference data; Comparing each of the angle differences with a second preset threshold value to obtain a plurality of comparison results; determining an error count of the reference data based on each of the comparison results; Returning to the step of selecting one of the pitch angles as reference data and using the remaining pitch angles as comparison data, until error counts corresponding to all the pitch angles are obtained; Based on the error count corresponding to each pitch angle, abnormal data is eliminated to obtain a plurality of normal pitch angles; Based on the multiple normal pitch angles, a weighted aggregation calculation is performed to obtain a pitch angle calibration result of the camera.
7. A camera calibration device, characterized in that: include: A shooting module, used to control the movement of the mobile robot and shoot the calibration object through a camera to obtain a first image set; the first image set includes images of the calibration object at different shooting distances and / or different shooting angles; A pitch angle calibration module, configured to calibrate the pitch angle of the camera based on the first image set, and determine a pitch angle calibration result of the camera; A control module, used for controlling the mobile robot to move to the front of the calibration object; The shooting module is further used to control the chassis to rotate, and to shoot the calibration object through the camera to obtain a second image set; the second image set includes images of the calibration object shot at different rotation angles; A position calibration module is used to calibrate the relative position of the camera and the chassis based on the second image set and the rotation angle information of the chassis, and determine the position calibration result of the camera.
8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the camera calibration method according to any one of claims 1 to 6 is implemented.
9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the camera calibration method according to any one of claims 1 to 6 is implemented.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the camera calibration method according to any one of claims 1 to 6 is implemented.