A camera extrinsic parameter self-calibration method and device

Abstract

The application discloses a camera external parameter self-calibration method and device. The camera external parameter is self-calibrated through a posture sensor and a high-precision positioning module, and the problem of increased positioning error caused by camera shaking or moving is solved. The posture sensor is used for correcting the change of the azimuth angle caused by camera shaking or moving, and the high-precision positioning module is used for eliminating the position offset error caused by shaking. Meanwhile, the BA algorithm is used to optimize the re-projection error, further optimize the camera external parameter, eliminate the posture sensor error and the high-precision positioning error, and finally realize the self-calibration of the camera external parameter. The application overcomes the problem that the traditional camera cannot adapt to camera shaking, rotation and offset, realizes the camera parameter self-adaptation, and expands the application scene of the camera.

Description

Technical Field

[0001] This invention belongs to the fields of intelligent transportation and machine vision technology, and in particular, it relates to a method and device for self-calibration of camera extrinsic parameters. Background Technology

[0002] Cameras, as a crucial means of roadside perception, play a vital role in intelligent transportation systems and vehicle-to-infrastructure (V2I) systems. By acquiring traffic video footage and performing structured analysis, they can detect, identify, and track traffic participants, thereby obtaining traffic flow and situational information within the road network. Simultaneously, they can also identify various traffic anomalies. Traditional cameras require calibration before extracting this traffic information, which involves establishing the mapping between camera image pixel coordinates and world coordinates.

[0003] Traditional cameras, once calibrated, have fixed calibration parameters that do not automatically adjust. In real-world scenarios, strong winds often occur, causing camera mounts to loosen and resulting in camera shake, rotation, and shifting. When the camera's position or orientation changes, traditional cameras need to be recalibrated before they can be used normally, increasing manual workload. Furthermore, they cannot adapt to real-time camera shake. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies by providing a camera extrinsic parameter self-calibration method and apparatus. This method overcomes the problem that traditional cameras cannot adapt to camera shake, rotation, and offset, proposes a camera self-calibration method, realizes camera parameter self-adaptation, and expands the application scenarios of the camera.

[0005] The technical solution to achieve the purpose of this invention is: a camera extrinsic parameter self-calibration method, the method comprising the following steps:

[0006] Calibrate the camera and attitude sensor, and obtain the initial extrinsic parameters of the camera and attitude sensor. The initial extrinsic parameters are obtained in the initial state, i.e., when the camera is fixed and there is no shaking. The attitude sensor extrinsic parameters are the rotation matrix from the attitude sensor coordinates to the world coordinates.

[0007] A first image frame is acquired and a first image feature is extracted, wherein the first image frame and the first image feature are obtained in the initial state;

[0008] Acquire first attitude data and first high-precision positioning data, wherein the first attitude data is the azimuth angle given by the attitude sensor in the initial state; and the first high-precision positioning data is the position vector given by the high-precision positioning module in the initial state.

[0009] Acquire a second image frame and extract second image features, wherein the second image frame and the second image features are obtained under conditions of camera shake or movement;

[0010] Acquire second attitude data, which is the attitude sensor azimuth angle corresponding to the second image frame data;

[0011] Calculate the rotation matrix corresponding to the second image frame;

[0012] Acquire the second high-precision positioning data, which is the high-precision positioning position vector corresponding to the second image frame data;

[0013] Calculate the translation matrix corresponding to the second image frame;

[0014] Calculate the camera extrinsic parameter matrix data corresponding to the second image frame;

[0015] By utilizing the first and second image features, the camera extrinsic matrix data is further optimized using the BA algorithm to complete the camera extrinsic self-calibration.

[0016] Furthermore, the step of acquiring the first image frame and extracting the first image features specifically includes:

[0017] In the initial state, N consecutive image frames are acquired to form the first image frame;

[0018] N sets of SIFT image features are extracted from the N image frames, consisting of K image feature points and K image feature vectors. Each set of SIFT image features corresponds one-to-one with an image frame. The first set of SIFT image features corresponds to the first image frame, the second set corresponds to the second image frame, and so on, up to the Nth set of SIFT image features, which corresponds to the Nth image frame. The first set of SIFT image features contains Q1 image feature points and Q1 image feature vectors, with each Q1 feature point and Q1 image feature vector corresponding to one another. The second set of SIFT image features contains Q2 image feature points and Q2 image feature vectors, with each Q2 feature point and Q2 image feature vector corresponding to one another, and so on, up to the Nth set of SIFT image features. N Image feature points and Q N A number of image feature vectors, Q N Image feature points and Q N Each image feature vector corresponds one-to-one;

[0019] M image feature points and M image feature vectors are extracted from the K image feature points and K image feature vectors, and the M image feature points and M image feature vectors constitute the first image feature.

[0020] Furthermore, the extraction of M image feature points and M image feature vectors specifically includes:

[0021] Perform pairwise matching on N sets of SIFT image features;

[0022] Count the number of times each feature point and its corresponding feature vector in the K image feature points and K image feature vectors are matched;

[0023] The number of times feature points and their corresponding feature vectors are matched is sorted from largest to smallest.

[0024] Select the first M image feature points and M image feature vectors.

[0025] Furthermore, the method for determining whether the camera is shaking or moving is as follows:

[0026] Acquire two consecutive frames of images;

[0027] Extract point features and line features from the image respectively;

[0028] Match point features and line features in two frames of images;

[0029] Calculate the image matching degree. If the matching degree is greater than a given threshold, it is determined that the camera has shaken or moved; otherwise, it is determined that the camera has not changed.

[0030] Furthermore, the method for extracting point features and line features from the image is as follows:

[0031] SIFT feature extraction is performed on the image to obtain feature points and feature vectors, which are used as the point features of the image;

[0032] Edge detection is performed on the image to obtain a binarized edge detection image;

[0033] A Hough transform is applied to the binarized edge detection image to obtain line features in the image; these line features are given by the following equation:

[0034] Γ={l i}

[0035]

[0036] Among them, l i This represents the i-th straight line in the line feature. Let k be the coordinates of any point on the i-th straight line. i It is a unit vector along the direction of the line.

[0037] Furthermore, the specific process for calculating the image matching degree includes:

[0038] Calculate the point feature matching degree:

[0039] m1 = n1 / N1

[0040] Where m1 is the point feature matching degree, n1 is the number of matched point features, and N1 is the total number of point features;

[0041] Calculate the line feature matching degree:

[0042] m2=n2 / N2

[0043] Where m2 is the line feature matching degree, n2 is the number of matched line features, and N2 is the total number of line features;

[0044] Calculate image matching degree:

[0045] m=α*m1+(1-α)*m2

[0046] Where m is the image matching degree and α is the matching factor.

[0047] Furthermore, the specific process of acquiring the second image frame and extracting the features of the second image includes:

[0048] When the camera shakes or moves, P consecutive image frames are acquired to form the second image frame;

[0049] P sets of SIFT image features are extracted from the P image frames, consisting of L image feature points and L image feature vectors. Each set of SIFT features corresponds one-to-one with an image frame. The first set of SIFT image features corresponds to the first image frame, the second set corresponds to the second image frame, and so on, up to the Pth set of SIFT image features, which corresponds to the Pth image frame. The first set of SIFT image features contains R1 image feature points and R1 image feature vectors, with each R1 image feature point and R1 image feature vector corresponding one-to-one. The second set of SIFT image features contains R2 image feature points and R2 image feature vectors, with each R2 image feature point and R2 image feature vector corresponding one-to-one, and so on, up to the Pth set of SIFT image features. P Image feature points and R P Image feature vectors, R P Image feature points and R P Each image feature vector corresponds one-to-one;

[0050] Q image feature points and Q image feature vectors are extracted from the L image feature points and L image feature vectors, and the Q image feature points and Q image feature vectors constitute the second image feature.

[0051] Furthermore, the specific process of calculating the rotation matrix corresponding to the second image frame includes:

[0052] The rotation matrix corresponding to the first pose is obtained using the first pose data, and is denoted as the first rotation matrix:

[0053]

[0054]

[0055]

[0056]

[0057] Where (θ1, φ1, ψ1) are the attitude angles of the camera relative to the attitude sensor coordinate system in the first attitude, and R1 is the first rotation matrix;

[0058] The rotation matrix corresponding to the second pose is obtained using the second pose data, and is denoted as the second rotation matrix:

[0059]

[0060]

[0061]

[0062]

[0063] Where (θ2, φ2, ψ2) are the attitude angles of the camera relative to the attitude sensor coordinate system in the second attitude, and R2 is the second rotation matrix;

[0064] In the attitude sensor coordinate system, calculate the rotation matrix of the second attitude relative to the first attitude, and denote it as the third rotation matrix:

[0065]

[0066] R3 is the third rotation matrix;

[0067] Projecting the third rotation matrix onto the world coordinate system, we obtain the fourth rotation matrix:

[0068]

[0069] Wherein, R4 is the fourth rotation matrix, and R0 is the attitude sensor extrinsic parameter matrix;

[0070] Furthermore, the specific process of calculating the translation matrix corresponding to the second image frame includes:

[0071] The camera's position coordinates in the first pose are obtained using the first high-precision positioning data, and are denoted as the first position coordinates:

[0072] P1 = (x1, y1, z1)

[0073] Where (x1, y1, z 1) represents the world coordinate system coordinates corresponding to the first high-precision positioning data, and P1 represents the coordinates of the first position;

[0074] The camera's position coordinates in the second pose are obtained using the second high-precision positioning data, and are denoted as the second position coordinates:

[0075] P2 = (x2, y2, z2)

[0076] Where (x2, y2, z2) are the world coordinates corresponding to the second high-precision positioning data, and P2 is the second position coordinate;

[0077] Calculate the translation matrix corresponding to the second image frame, denoted as the second translation matrix T2:

[0078]

[0079] The camera extrinsic matrix corresponding to the second image frame, denoted as the second camera extrinsic matrix, is given by the following equation:

[0080]

[0081]

[0082] Wherein, M0 is the initial extrinsic parameter matrix of the camera, R and T are the rotation matrix and translation matrix in the initial extrinsic parameter matrix M0, respectively, M2 is the second extrinsic parameter matrix of the camera, R4 is the fourth rotation matrix, and T2 is the second translation matrix.

[0083] Furthermore, the process of further optimizing the camera extrinsic parameter matrix data using the first image features and the second image features through the BA algorithm includes:

[0084] Match the features of the first image and the features of the second image to obtain matching point pairs;

[0085] Construct the world coordinates corresponding to the matching point pairs as optimization parameters;

[0086] Using the camera intrinsic data, the initial extrinsic matrix, and the second camera extrinsic matrix, the world coordinates corresponding to the matching point pairs are projected onto the first pose and the second pose image coordinate systems, respectively.

[0087] Calculate the projection coordinate error of the first attitude and the projection coordinate error of the second attitude respectively;

[0088] With fixed camera intrinsic data and initial extrinsic matrix, the BA algorithm is used to optimize the extrinsic matrix of the second camera.

[0089] A camera extrinsic parameter self-calibration device, the device comprising:

[0090] The image acquisition unit is used to acquire video images;

[0091] The attitude calculation unit is used to acquire camera attitude data;

[0092] High-precision positioning unit, used to acquire high-precision positioning data from the camera;

[0093] The computing unit is used to calculate the first image features and the second image features;

[0094] The computing unit is also used to calculate the rotation matrix and translation matrix corresponding to the second image frame;

[0095] The computing unit is also used to calculate the camera extrinsic parameters corresponding to the second image frame;

[0096] The computing unit is also used in the BA algorithm to optimize camera extrinsic parameters.

[0097] Compared with the prior art, the present invention has the following significant advantages:

[0098] 1) Traditionally, camera extrinsic and extrinsic parameters are calibrated only during initial use and remain fixed thereafter. When the camera's position or attitude changes, the calibration parameters become invalid, and the camera needs to be recalibrated. This invention proposes a self-calibration method for camera extrinsic parameters. It automatically acquires changes in camera position and attitude using attitude and positioning sensors, and then automatically adjusts the camera's extrinsic parameters without manual intervention or recalibration, greatly reducing workload and broadening the applicable scenarios for the camera.

[0099] 2) The present invention proposes a method for automatically identifying camera shake or movement. By extracting image point features and line features, the robustness of the method is enhanced. The extrinsic parameter self-calibration algorithm is only performed when a change in camera position or attitude is detected, which reduces the amount of computation and lowers the power consumption of the device.

[0100] The present invention will now be described in further detail with reference to the accompanying drawings. Attached Figure Description

[0101] Figure 1 This is a flowchart of a camera extrinsic parameter self-calibration method provided in an embodiment of the present invention.

[0102] Figure 2 This is a schematic diagram illustrating the relationship between different coordinate systems provided in an embodiment of the present invention.

[0103] Figure 3 This is a schematic diagram of the attitude calculation unit provided in an embodiment of the present invention.

[0104] Figure 4 This is a schematic diagram of a high-precision positioning unit provided in an embodiment of the present invention.

[0105] Figure 5 This is a schematic diagram illustrating the first implementation method of the camera extrinsic parameter calibration device provided in an embodiment of the present invention.

[0106] Figure 6 This is a schematic diagram illustrating a second method for implementing a camera extrinsic parameter calibration device according to an embodiment of the present invention. Detailed Implementation

[0107] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0108] It should be noted that if the embodiments of the present invention involve descriptions such as "first" and "second," these descriptions are for descriptive purposes only and should not be construed as indicating or implying their relative importance or implicitly specifying the number of technical features indicated. Therefore, a feature defined with "first" and "second" may explicitly or implicitly include at least one of those features. Furthermore, the technical solutions of the various embodiments can be combined with each other, but this must be based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or impossible to implement, it should be considered that such a combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0109] In one embodiment, a camera extrinsic parameter self-calibration method is provided. This method acquires changes in camera attitude and position using attitude and position sensors, automatically calculates camera extrinsic parameter data, and achieves the purpose of camera extrinsic parameter self-calibration. For example... Figure 1 As shown. It should be noted that the image coordinates are the pixel coordinates in the image, two-dimensional vectors. World coordinates are the three-dimensional coordinates of the actual object corresponding to the image pixels, which can be UTM coordinates, ENU coordinates, or a custom coordinate system. The world coordinates used in this application are UTM coordinates, obtained by converting GPS coordinates from the position sensor. The specific calibration steps are as follows:

[0110] Step 1: Calibrate the camera and attitude sensor;

[0111] This step is to obtain the camera's intrinsic and extrinsic parameter data and the attitude sensor's extrinsic parameter data during initial installation. The initial parameters are obtained in the initial state, i.e., when the camera is fixed and there is no shaking. The attitude sensor's extrinsic parameter is the rotation matrix from the attitude sensor's own coordinate system to the world coordinate system. If the attitude sensor has been calibrated with the actual coordinate system, then this extrinsic parameter is the identity matrix. The world coordinate system used during camera extrinsic parameter calibration must be consistent with the world coordinate system used in high-precision positioning.

[0112] Step 2: Acquire the first image frame and extract the first image features. The first image frame is acquired in the initial state, i.e., when the camera has not moved or rotated. The specific steps are as follows:

[0113] Step 2-1: In the initial state, acquire N consecutive image frames to form the first image frame;

[0114] Step 2-2: Extract SIFT image features from N image frames respectively to obtain all image feature points and corresponding feature vectors; specifically: extract N sets of SIFT image features from the N image frames, totaling K image feature points and K image feature vectors. Each set of SIFT image features corresponds one-to-one with an image frame. The first set of SIFT image features corresponds to the first image frame, the second set corresponds to the second image frame, ..., and the Nth set of SIFT image features corresponds to the Nth image frame; the first set of SIFT image features contains Q1 image feature points and Q1 image feature vectors, with each Q1 image feature point and Q1 image feature vector corresponding one-to-one; the second set of SIFT image features contains Q2 image feature points and Q2 image feature vectors, with each Q2 image feature point and Q2 image feature vector corresponding one-to-one, ..., and the Nth set of SIFT image features contains Q1 image feature points and Q1 image feature vectors, with each Q1 image feature point and Q1 image feature vector corresponding one-to-one; the second set of SIFT image features contains Q1 image feature points and Q2 image feature vectors, with each Q2 image feature point and Q2 image feature vector corresponding one-to-one, ..., and the Nth set of SIFT image features contains Q1 image feature points and Q1 image feature vectors, with each Q1 image feature point and Q1 image feature vector corresponding one-to-one; ... N Image feature points and Q N A number of image feature vectors, Q N Image feature points and Q N Each image feature vector corresponds one-to-one;

[0115] Steps 2-3: Perform pairwise matching on the SIFT features extracted from the N image frames;

[0116] Steps 2-4: Count the number of times all image feature points and their corresponding feature vectors are matched;

[0117] Steps 2-5: Sort the matching counts from largest to smallest;

[0118] Steps 2-6: Select the M image feature points with the highest matching frequency and their corresponding image feature vectors to obtain the first image feature.

[0119] Step 3: Acquire the first attitude data and the first high-precision positioning data;

[0120] For example, in one embodiment, the first attitude data corresponds to the azimuth angle given by the attitude sensor, and the first high-precision positioning data corresponds to the UTM coordinates obtained by converting the GPS coordinates given by the position sensor.

[0121] Step 4: Acquire the second image frame and extract the features of the second image;

[0122] The second image frame is acquired when a change in camera position or pose is detected; it is not necessary to acquire when the camera position or pose has not changed. The steps for detecting a change in camera position or pose are as follows:

[0123] Step 4-1: Acquire two consecutive frames of images;

[0124] Step 4-2: Extract point features and line features from the image respectively;

[0125] Step 4-3: Match point features and line features in the two frames of images;

[0126] Step 4-4: Calculate the image matching degree and determine whether the camera position or pose has changed. If the matching degree is greater than a given threshold, it is determined that the camera has changed; otherwise, it is determined that the camera has not changed.

[0127] For example, in one embodiment, SIFT features of the image are selected as point features, and line features are obtained by edge detection of the image. The line features are described by the following linear equation:

[0128] Γ={l i}

[0129] Among them, l i This represents the i-th straight line in the line feature. Let k be the coordinates of any point on the i-th straight line. i It is a unit vector along the direction of the line.

[0130] For example, in one embodiment, the total matching degree is calculated as a weighted average of the point feature matching degree and the line feature matching degree. Specifically:

[0131] Calculate the point feature matching degree:

[0132] m1 = n1 / N1

[0133] Where m1 is the point feature matching degree, n1 is the number of matched point features, and N1 is the total number of point features;

[0134] Calculate the line feature matching degree:

[0135] m2=n2 / N2

[0136] Where m2 is the line feature matching degree, n2 is the number of matched line features, and N2 is the total number of line features;

[0137] Calculate image matching degree:

[0138] m=α*m1+(1-α)*m2

[0139] Where m is the image matching degree and α is the matching factor.

[0140] For example, in one embodiment, the extraction of the second image features uses the same method as the extraction of the first image features.

[0141] Step 5: Acquire the second attitude data and the second high-precision positioning data;

[0142] For example, in one embodiment, the second attitude data corresponds to the azimuth angle given by the attitude sensor, and the second high-precision positioning data corresponds to the UTM coordinates obtained by converting the GPS coordinates given by the position sensor.

[0143] Step 6: Calculate the rotation matrix and translation matrix corresponding to the second image frame;

[0144] Figure 2 The correspondence between different coordinate systems is given. The rotation matrix corresponding to the second image frame is the rotation transformation of the camera pose corresponding to the second image frame relative to the camera pose corresponding to the first image frame in the world coordinate system. The azimuth angle given by the attitude sensor is obtained in the attitude sensor's own coordinate system. The calculation of the rotation matrix requires the simultaneous use of attitude sensor data and initial attitude sensor extrinsic parameters. The calculation steps are as follows:

[0145] Step 6-1: Obtain the rotation matrix corresponding to the first attitude using the first attitude data, denoted as the first rotation matrix, which is given by the following equation:

[0146]

[0147]

[0148]

[0149]

[0150] Where (θ1, φ1, ψ1) are the attitude angles of the camera relative to the attitude sensor coordinate system in the first attitude, and R1 is the first rotation matrix;

[0151] Step 6-2: Obtain the rotation matrix corresponding to the second attitude using the second attitude data, denoted as the second rotation matrix, which is given by the following equation:

[0152]

[0153]

[0154]

[0155]

[0156] Where (θ2, φ2, ψ2) are the attitude angles of the camera relative to the attitude sensor coordinate system in the second attitude, and R2 is the second rotation matrix;

[0157] Step 6-3: In the attitude sensor coordinate system, calculate the rotation matrix of the second attitude relative to the first attitude, denoted as the third rotation matrix, given by the following equation:

[0158]

[0159] Wherein, R3 is the third rotation matrix;

[0160] Step 6-4: Project the third rotation matrix onto the world coordinate system to obtain the fourth rotation matrix, given by the following equation:

[0161]

[0162] Wherein, R4 is the fourth rotation matrix, and R0 is the attitude sensor extrinsic parameter matrix;

[0163] The translation matrix is ​​given by the following equation:

[0164] P1 = (x1, y1, z1)

[0165] P2 = (x2, y2, z2)

[0166] Where (x1, y1, z1) are the world coordinates corresponding to the first high-precision positioning data, (x2, y2, z2) are the world coordinates corresponding to the second high-precision positioning data, and T2 is the second translation matrix.

[0167] Step 7: Calculate the camera extrinsic data corresponding to the second image frame. Here, the extrinsic calibration values ​​under camera shake are obtained from the attitude sensor and position sensor. This value is only an initial value for the camera extrinsic parameters, and further optimization algorithms are needed to obtain more accurate extrinsic parameter values. The camera extrinsic parameter matrix is ​​given by the following equation:

[0168]

[0169]

[0170] Wherein, M0 is the initial extrinsic parameter matrix of the camera, R and T are the rotation matrix and translation matrix in the initial extrinsic parameter matrix M0, respectively, M2 is the second extrinsic parameter matrix of the camera, R4 is the fourth rotation matrix, and T2 is the second translation matrix.

[0171] Step 8: Utilize the first and second image features to further optimize the camera extrinsic data using the BA algorithm. The specific steps are as follows:

[0172] Step 8-1: Match the features of the first image and the features of the second image to obtain matching point pairs;

[0173] Step 8-2: Construct the world coordinates corresponding to the matching point pairs as optimization parameters;

[0174] Step 8-3: Using the camera intrinsic data, the initial extrinsic matrix, and the second extrinsic matrix, project the world coordinates corresponding to the matching point pairs onto the first pose and the second pose image coordinate systems, respectively.

[0175] Step 8-4: Calculate the projection coordinate error of the first attitude and the projection coordinate error of the second attitude respectively;

[0176] Step 8-5: Fix the camera intrinsic data and the initial extrinsic matrix, and use the BA algorithm to optimize the second extrinsic matrix.

[0177] Step 9: Complete the self-calibration of camera extrinsic parameters.

[0178] In one embodiment, a camera extrinsic parameter self-calibration device is provided, including an image acquisition unit, an attitude calculation unit, a high-precision positioning unit, and a calculation unit. The image acquisition unit is used to acquire video images; the attitude calculation unit is used to acquire camera attitude data; the high-precision positioning unit is used to acquire camera high-precision positioning data; the calculation unit is used to calculate first image features and second image features; the calculation unit is also used to calculate the rotation matrix and translation matrix corresponding to the second image frame; the calculation unit is also used to calculate the camera extrinsic parameters corresponding to the second image frame; and the calculation unit is also used in a BA algorithm to optimize the camera extrinsic parameters.

[0179] One possible implementation is that the attitude calculation unit consists of a gyroscope, an angular velocity meter, a geomagnetic sensor, and a calculation unit, such as... Figure 3 As shown.

[0180] Other possible implementations include a high-precision positioning unit consisting of a GPS module, a ground-based augmentation module, and a positioning unit, such as... Figure 4 As shown.

[0181] Other possible implementations include integrating the image acquisition unit, attitude calculation unit, high-precision positioning unit, and computing unit into a single device, such as... Figure 5 As shown, the device can be a camera device.

[0182] Other possible implementations include having the image acquisition unit, attitude calculation unit, high-precision positioning unit, and computing unit located in different devices, such as... Figure 6As shown in the diagram. The image acquisition unit, attitude calculation unit, and high-precision positioning unit can be located within the camera device, while the computing unit can be located within the edge computing device. The two can transmit data through a communication unit, and a single edge computing device can connect to one or more camera devices.

[0183] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to perform the following steps:

[0184] Calibrate the camera and attitude sensor, and obtain the initial extrinsic parameters of the camera and attitude sensor. The initial extrinsic parameters are obtained in the initial state, i.e., when the camera is fixed and there is no shaking. The attitude sensor extrinsic parameters are the rotation matrix from the attitude sensor coordinates to the world coordinates.

[0185] A first image frame is acquired and a first image feature is extracted, wherein the first image frame and the first image feature are obtained in the initial state;

[0186] Acquire first attitude data and first high-precision positioning data, wherein the first attitude data is the azimuth angle given by the attitude sensor in the initial state; and the first high-precision positioning data is the position vector given by the high-precision positioning module in the initial state.

[0187] Acquire a second image frame and extract second image features, wherein the second image frame and the second image features are obtained under conditions of camera shake or movement;

[0188] Acquire second attitude data, which is the attitude sensor azimuth angle corresponding to the second image frame data;

[0189] Calculate the rotation matrix corresponding to the second image frame;

[0190] Acquire the second high-precision positioning data, which is the high-precision positioning position vector corresponding to the second image frame data;

[0191] Calculate the translation matrix corresponding to the second image frame;

[0192] Calculate the camera extrinsic parameter matrix data corresponding to the second image frame;

[0193] By utilizing the first and second image features, the camera extrinsic matrix data is further optimized using the BA algorithm to complete the camera extrinsic self-calibration.

[0194] For specific limitations on each step, please refer to the limitations on the camera extrinsic parameter self-calibration method mentioned above, which will not be repeated here.

[0195] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, the computer program performing the following steps when executed by a processor:

[0196] Calibrate the camera and attitude sensor, and obtain the initial extrinsic parameters of the camera and attitude sensor. The initial extrinsic parameters are obtained in the initial state, i.e., when the camera is fixed and there is no shaking. The attitude sensor extrinsic parameters are the rotation matrix from the attitude sensor coordinates to the world coordinates.

[0197] A first image frame is acquired and a first image feature is extracted, wherein the first image frame and the first image feature are obtained in the initial state;

[0198] Acquire first attitude data and first high-precision positioning data, wherein the first attitude data is the azimuth angle given by the attitude sensor in the initial state; and the first high-precision positioning data is the position vector given by the high-precision positioning module in the initial state.

[0199] Acquire a second image frame and extract second image features, wherein the second image frame and the second image features are obtained under conditions of camera shake or movement;

[0200] Acquire second attitude data, which is the attitude sensor azimuth angle corresponding to the second image frame data;

[0201] Calculate the rotation matrix corresponding to the second image frame;

[0202] Acquire the second high-precision positioning data, which is the high-precision positioning position vector corresponding to the second image frame data;

[0203] Calculate the translation matrix corresponding to the second image frame;

[0204] Calculate the camera extrinsic parameter matrix data corresponding to the second image frame;

[0205] By utilizing the first and second image features, the camera extrinsic matrix data is further optimized using the BA algorithm to complete the camera extrinsic self-calibration.

[0206] For specific limitations on each step, please refer to the limitations on the camera extrinsic parameter self-calibration method mentioned above, which will not be repeated here.

[0207] The foregoing has shown and described the basic principles, main features, and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited to the above embodiments. The embodiments and descriptions in the specification are merely illustrative of the principles of the invention. Any modifications, equivalent substitutions, or improvements made within the spirit and principles of the present invention without departing from its spirit and scope should be included within the protection scope of the present invention.

Claims

1. A method for self-calibrating camera extrinsic parameters, characterized in that, The method includes the following steps: Calibrate the camera and attitude sensor, and obtain the initial extrinsic parameters of the camera and attitude sensor. The initial extrinsic parameters are obtained in the initial state, i.e., when the camera is fixed and there is no shaking. The attitude sensor extrinsic parameters are the rotation matrix from the attitude sensor coordinates to the world coordinates. A first image frame is acquired and a first image feature is extracted, wherein the first image frame and the first image feature are obtained in the initial state; Acquire first attitude data and first high-precision positioning data, wherein the first attitude data is the azimuth angle given by the attitude sensor in the initial state; and the first high-precision positioning data is the position vector given by the high-precision positioning module in the initial state. Acquire a second image frame and extract second image features, wherein the second image frame and the second image features are obtained under conditions of camera shake or movement; Acquire second attitude data, which is the attitude sensor azimuth angle corresponding to the second image frame data; Calculate the rotation matrix corresponding to the second image frame; Acquire the second high-precision positioning data, which is the high-precision positioning position vector corresponding to the second image frame data; Calculate the translation matrix corresponding to the second image frame; Calculate the camera extrinsic parameter matrix data corresponding to the second image frame; Using the first and second image features, the camera extrinsic matrix data is further optimized through the BA algorithm to complete the camera extrinsic self-calibration. The specific process for calculating the rotation matrix corresponding to the second image frame includes: The rotation matrix corresponding to the first pose is obtained using the first pose data, and is denoted as the first rotation matrix: Where (θ1,φ1,ψ1) is the attitude angle of the camera relative to the attitude sensor coordinate system in the first attitude, and R1 is the first rotation matrix; The rotation matrix corresponding to the second pose is obtained using the second pose data, and is denoted as the second rotation matrix: Where (θ2,φ2,ψ2) is the camera's attitude angle relative to the attitude sensor coordinate system in the second attitude, and R2 is the second rotation matrix; In the attitude sensor coordinate system, calculate the rotation matrix of the second attitude relative to the first attitude, and denote it as the third rotation matrix: R3 is the third rotation matrix; Projecting the third rotation matrix onto the world coordinate system, we obtain the fourth rotation matrix: Wherein, R4 is the fourth rotation matrix, and R0 is the attitude sensor extrinsic parameter matrix; The specific process for calculating the translation matrix corresponding to the second image frame includes: The camera's position coordinates in the first pose are obtained using the first high-precision positioning data, and are denoted as the first position coordinates: P1 = (x1, y1, z1) where (x1, y1, z1) are the world coordinates corresponding to the first high-precision positioning data, and P1 is the first position coordinate; The camera's position coordinates in the second pose are obtained using the second high-precision positioning data, and are denoted as the second position coordinates: P2 = (x2, y2, z2) where (x2, y2, z2) are the world coordinates corresponding to the second high-precision positioning data, and P2 is the second position coordinate; Calculate the translation matrix corresponding to the second image frame, denoted as the second translation matrix T2: The camera extrinsic matrix corresponding to the second image frame, denoted as the second camera extrinsic matrix, is given by the following equation: Wherein, M0 is the initial extrinsic parameter matrix of the camera, R and T are the rotation matrix and translation matrix in the initial extrinsic parameter matrix M0, respectively, M2 is the second extrinsic parameter matrix of the camera, R4 is the fourth rotation matrix, and T2 is the second translation matrix.

2. The camera extrinsic parameter self-calibration method according to claim 1, characterized in that, The process of acquiring the first image frame and extracting the first image features specifically includes: In the initial state, N consecutive image frames are acquired to form the first image frame; N sets of SIFT image features are extracted from the N image frames, consisting of K image feature points and K image feature vectors. Each set of SIFT image features corresponds one-to-one with an image frame. The first set of SIFT image features corresponds to the first image frame, the second set corresponds to the second image frame, and so on, up to the Nth set of SIFT image features, which corresponds to the Nth image frame. The first set of SIFT image features contains Q1 image feature points and Q1 image feature vectors, with each Q1 feature point and Q1 image feature vector corresponding to one another. The second set of SIFT image features contains Q2 image feature points and Q2 image feature vectors, with each Q2 feature point and Q2 image feature vector corresponding to one another, and so on, up to the Nth set of SIFT image features. N Image feature points and Q N A number of image feature vectors, Q N Image feature points and Q N Each image feature vector corresponds one-to-one; M image feature points and M image feature vectors are extracted from the K image feature points and K image feature vectors, and the M image feature points and M image feature vectors constitute the first image feature.

3. The camera extrinsic parameter self-calibration method according to claim 2, characterized in that, The extraction of M image feature points and M image feature vectors specifically includes: Perform pairwise matching on N sets of SIFT image features; Count the number of times each feature point and its corresponding feature vector in the K image feature points and K image feature vectors are matched; The number of times feature points and their corresponding feature vectors are matched is sorted from largest to smallest. Select the first M image feature points and M image feature vectors.

4. The camera extrinsic parameter self-calibration method according to claim 1, characterized in that, The method for determining whether the camera is shaking or moving is as follows: Acquire two consecutive frames of images; Extract point features and line features from the image respectively; Match point features and line features in two frames of images; Calculate the image matching degree. If the matching degree is greater than a given threshold, it is determined that the camera has shaken or moved; otherwise, it is determined that the camera has not changed.

5. The camera extrinsic parameter self-calibration method according to claim 4, characterized in that, The method for extracting point and line features from an image is as follows: SIFT feature extraction is performed on the image to obtain feature points and feature vectors, which are used as the point features of the image; Edge detection is performed on the image to obtain a binarized edge detection image; A Hough transform is applied to the binarized edge detection image to obtain line features in the image; these line features are given by the following equation: C={l i } Among them, l i This represents the i-th straight line in the line feature. Let k be the coordinates of any point on the i-th straight line. i It is a unit vector along the direction of the line.

6. The camera extrinsic parameter self-calibration method according to claim 4, characterized in that, The specific process for calculating the image matching degree includes: Calculate the point feature matching degree: m1 = n1 / N1 Where m1 is the point feature matching degree, n1 is the number of matched point features, and N1 is the total number of point features; Calculate the line feature matching degree: m2=n2 / N2 Where m2 is the line feature matching degree, n2 is the number of matched line features, and N2 is the total number of line features; Calculate image matching degree: m=α*m1+(1-α)*m2 Where m is the image matching degree and α is the matching factor.

7. The camera extrinsic parameter self-calibration method according to claim 1, characterized in that, The specific process of acquiring the second image frame and extracting the features of the second image includes: When the camera shakes or moves, P consecutive image frames are acquired to form the second image frame; P sets of SIFT image features are extracted from the P image frames, consisting of L image feature points and L image feature vectors. Each set of SIFT features corresponds one-to-one with an image frame. The first set of SIFT image features corresponds to the first image frame, the second set corresponds to the second image frame, and so on, up to the Pth set of SIFT image features, which corresponds to the Pth image frame. The first set of SIFT image features contains R1 image feature points and R1 image feature vectors, with each R1 image feature point and R1 image feature vector corresponding one-to-one. The second set of SIFT image features contains R2 image feature points and R2 image feature vectors, with each R2 image feature point and R2 image feature vector corresponding one-to-one, and so on, up to the Pth set of SIFT image features. P Image feature points and R P R is a set of image feature vectors. P Image feature points and R P Each image feature vector corresponds one-to-one; Q image feature points and Q image feature vectors are extracted from the L image feature points and L image feature vectors, and the Q image feature points and Q image feature vectors constitute the second image feature.

8. The camera extrinsic parameter self-calibration method according to claim 1, characterized in that, The process of further optimizing the camera extrinsic parameter matrix data using the first and second image features and the BA algorithm includes: Match the features of the first image and the features of the second image to obtain matching point pairs; Construct the world coordinates corresponding to the matching point pairs as optimization parameters; Using the camera intrinsic data, the initial extrinsic matrix, and the second camera extrinsic matrix, the world coordinates corresponding to the matching point pairs are projected onto the first pose and the second pose image coordinate systems, respectively. Calculate the projection coordinate error of the first attitude and the projection coordinate error of the second attitude respectively; With fixed camera intrinsic data and initial extrinsic matrix, the BA algorithm is used to optimize the extrinsic matrix of the second camera.

9. A camera extrinsic parameter self-calibration device based on the method of any one of claims 1 to 8, characterized in that, The device includes an image acquisition unit, an attitude calculation unit, a high-precision positioning unit, and a calculation unit; wherein, the image acquisition unit is used to acquire video images; the attitude calculation unit is used to acquire camera attitude data; the high-precision positioning unit is used to acquire camera high-precision positioning data; the calculation unit is used to calculate first image features and second image features; the calculation unit is also used to calculate the rotation matrix and translation matrix corresponding to the second image frame; the calculation unit is also used to calculate the camera extrinsic parameters corresponding to the second image frame; the calculation unit is also used for the BA algorithm to optimize the camera extrinsic parameters.

10. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes a computer program, it implements the method as described in any one of claims 1 to 8.

Images