Vanishing point based camera gravity direction self-correction method

Abstract

The application provides a camera gravity direction self-correction method based on vanishing points to restore the gravity direction of the camera during shooting. Vehicle driving record data is used to extract and calculate vertical line segments in the image, and the roll angle and the pitch angle of the image are calculated according to a single-hole camera imaging model, and the camera gravity direction posture is restored without gravity data. The method is simple and fast, and can quickly extract the roll angle and the pitch angle posture of the camera during shooting, and provides reliable data support for information extraction and utilization of the driving record data.

Description

Technical Field

[0001] This invention provides a camera gravity direction self-correction method based on vanishing point, belonging to the field of image processing technology. Background Technology

[0002] Dashcam image data is a multi-dimensional information sequence of images and corresponding location data of vehicles recorded at regular time intervals. It is a crucial component of numerous road traffic crowdsourced data sources, containing rich semantic information about roads. In applications such as high-precision map updates and traffic monitoring, crowdsourced data sources like dashcam data play a key role and possess potential for information mining. These data sources provide valuable information by capturing real-time vehicle movement and interactions on the road, which is essential for maintaining and updating high-precision maps and achieving accurate traffic monitoring.

[0003] However, the utilization of sequential image data, such as dashcam footage, is currently quite limited. This type of data mainly comes from devices such as mobile phones and dashcams. Due to the limited sensor information recorded and the low frame rate, it is difficult to extract information and reconstruct the camera's pose at the time of shooting using existing SLAM algorithms. Therefore, analyzing and utilizing the information in this data presents certain challenges. Summary of the Invention

[0004] To overcome this difficulty, this invention proposes a camera gravity direction self-correction method based on the vanishing point to recover the camera's gravity direction during shooting. This method is simple and fast, and can quickly extract the camera's roll and pitch angles during shooting, providing reliable data support for the extraction and utilization of information from driving recording data.

[0005] This invention uses vehicle driving record data, extracts and calculates vertical line segments within the image, and calculates the roll and pitch angles of the image based on a single-aperture camera imaging model, thereby restoring the camera's gravity orientation in the absence of gravity data.

[0006] To calculate the direction of gravity in a single image, the specific technical solution of this invention is as follows:

[0007] S1. Extraction of features of line segments in the image;

[0008] Based on the input image, line segments in the image are identified through line detection, and vertical line segments are selected.

[0009] Image line segment feature extraction is achieved through algorithms such as LSD fast line detection.

[0010] S2. Extraction of vertical line segment features from the image

[0011] By setting a threshold for the rotation angle of the line segment relative to the y-direction of the image, a batch of line segments that meet the slope threshold are selected from the line segment extraction results. At the same time, there are many fragmented line segments in the extraction results, so the length of the line segments also needs to be selected. The final selection result is used as the spatial vertical line segment result for calculating the vanishing point position.

[0012] S3, Calculation of image gravity direction

[0013] S3.1 Image Gravity Direction Calculation Method

[0014] Suppose there are two vertical line segments AB and CD in two segments of space, and the spatial coordinates of the four points ABCD are (X, X, C, D) respectively. A Y A Z A ),(X B Y B Z B ),(X C Y C Z C ) and (X D Y D Z D Let P be the intersection point of two spatial vertical lines in the image plane, with coordinates (dx, dy).

[0015] Let the extrinsic parameters of the camera be R and T, where R represents the rotation matrix between the camera coordinate system and the spatial coordinate system, and T represents the translation matrix between the two coordinate systems. Since the current calculation only involves rotation, T = [0 0 0]. T Suppose that the components in R are as follows:

[0016]

[0017] The coordinates (X, Y, Z) of a spatial point and the coordinates (x, y) of an image point on the photograph are collinear, where (X, Y, Z) = (x, y) / ( ... S Y S Z S Let ) be the coordinates of the camera in the spatial coordinate system:

[0018]

[0019] Substituting the coordinates of the four points, we obtain the equations of the lines projected onto the image plane by the two spatial lines:

[0020] Line AB: x = m AB ·y+b AB

[0021] Line CD: x = m CD ·y+b CD

[0022] The coordinates of the intersection point P of AB and CD are calculated as follows: This coordinate is independent of the spatial point coordinates, and only depends on the camera's spatial rotation attitude; therefore, when the image plane has a certain angle with the vertical direction, any vertical line in space will intersect at a point in the image plane, and this point is the vanishing point of this set of parallel lines in the current image; since T=[0 0 0] T Camera coordinate system X C With the three-dimensional coordinate system X W The conversion relationship between them is as follows:

[0023] X C =RX W

[0024] When only considering the camera's pitch and roll changes—that is, the camera first rotates ω around the X-axis (pitch transformation) and then rotates β around the Z-axis (roll transformation)—the image plane coordinates of the vanishing point can be obtained. Its coordinates in the three-dimensional coordinate system are That is, the intersection of the vertical line passing through the center of the camera projection and the image plane in the three-dimensional coordinate system;

[0025] By extracting the vertical terrain contours, the vanishing point coordinates are calculated, and the roll and pitch angles are then calculated using these coordinates. The coordinates of the vanishing point in the image plane are (dx, dy). The pitch and roll transformation angles ω and β from the spatial 3D coordinate system to the camera coordinate system are calculated, ignoring orientation.

[0026]

[0027]

[0028] S3.2 Vertical Vanishing Point Calculation

[0029] In the set of vertical line segments obtained in step S2, the coordinates of the intersection point between any two line segments can be calculated, which is the observation value of the vanishing point in the vertical direction of the image; the product of the lengths of the two line segments is used as the weight of the observation point to obtain the weighted distribution of the vanishing point observation value.

[0030] S3.3 Vertical Vanishing Point Cluster Center Calculation

[0031] DBSCAN is used to cluster high-weight vanishing points and the coordinates of the cluster centers are calculated to obtain optimized vanishing point measurement results.

[0032] S4. Gravity Direction Calculation and Restoration

[0033] Substitute the result of the vanishing point position calculation into the formula in step S3.1 to calculate the camera pitch angle and roll angle, and use these angles to recover the image. Attached Figure Description

[0034] Figure 1 The driving recorder data test image is shown in the example.

[0035] Figure 2 Linear features in the image of the example;

[0036] Figure 3 The results of extracting vertical line segments in the example;

[0037] Figure 4 This is a schematic diagram illustrating the principle of gravity direction calculation in an embodiment.

[0038] Figure 5 This is a distribution map of vanishing point observations for an example.

[0039] Figure 6 The vertical vanishing point clustering results and position calculation results are shown in the example.

[0040] Figure 7 The image shows the restored gravity direction as an example. Detailed Implementation

[0041] A camera gravity orientation self-correction method based on vanishing point includes the following steps:

[0042] S1. Extraction of features of line segments in the image;

[0043] Based on the input image, it is necessary to identify line segments in the image through line detection and filter out vertical line segments. In urban environments, the scenery in dashcam images mainly consists of buildings, bridges, streetlights, etc., which are generally constructed and installed vertically. Therefore, line segments along the y-direction in the image have a high probability of being vertical line segments. Figure 1 The example dashcam image shows buildings and road signs with their edges pointing vertically.

[0044] Image line segment feature extraction can be achieved using algorithms such as LSD fast line detection, and the extraction results are as follows: Figure 2 As shown.

[0045] S2. Extraction of vertical line segment features from the image

[0046] Since camera installation inevitably involves pitch and roll angles, although these angles are relatively small, the slope change of the image formed by vertical line segments in space on the image plane is generally not significant. Therefore, a threshold for the rotation angle of line segments relative to the image's y-direction can be set to filter a batch of line segments from the extracted results that meet the slope threshold. Simultaneously, many fragmented line segments exist in the extracted results, necessitating further filtering by segment length. The final filtered result is then used as the spatial vertical line segment result, such as... Figure 3This is used to calculate the vanishing point location.

[0047] S3, Calculation of image gravity direction

[0048] Step 3 is the core step of this invention.

[0049] S3.1 Image Gravity Direction Calculation Principle

[0050] Assumption Figure 3 In a given space, there are two vertical line segments AB and CD, and the spatial coordinates of points ABCD are (X, X, C, D) respectively. A Y A Z A ),(X B Y B Z B ),(X C Y C Z C ) and (X D Y D Z D Let P be the intersection point of two spatial vertical lines in the image plane, with coordinates (dx, dy). Figure 4 As shown.

[0051] Assume the camera's extrinsic parameters are R and T, where R represents the rotation matrix between the camera coordinate system and the spatial coordinate system, and T represents the translation matrix between the two coordinate systems. Since the current calculation only involves rotation, T = [0 0 0]. T Suppose that the components in R are as follows:

[0052]

[0053] According to the principle that light travels in straight lines, the coordinates (X, Y, Z) of a spatial point and the coordinates (x, y) of an image point on a photograph are collinear, where (X, Y, Z) = (x, y) / (x, y). S Y S Z S Let ) be the coordinates of the camera in the spatial coordinate system:

[0054]

[0055] Substituting the coordinates of the four points, we obtain the equations of the lines projected onto the image plane by the two spatial lines:

[0056] Line AB: x = m AB ·y+b AB

[0057] Line CD: x = m CD ·y+b CD

[0058] The coordinates of the intersection point P of AB and CD are calculated as follows: This coordinate is independent of the spatial point coordinates, and only depends on the camera's spatial rotation attitude. Therefore, when the image plane has a certain angle with the vertical direction, any vertical line in space will intersect at a single point in the image plane; this point is the vanishing point of this set of parallel lines in the current image. Since T = [0 0 0] T Camera coordinate system X C With the three-dimensional coordinate system X W The conversion relationship between them is as follows:

[0059] X C =RX W

[0060] When only considering the camera's pitch and roll changes, i.e., the camera first rotates ω around the X-axis (pitch transformation) and then rotates β around the Z-axis (roll transformation), the image plane coordinates of the vanishing point can be obtained. Its coordinates in the three-dimensional coordinate system are It is the intersection of the vertical line passing through the center of the camera projection and the image plane in the three-dimensional coordinate system.

[0061] Therefore, for a single photograph, the roll and pitch angles can be calculated by extracting the outlines of vertical features, calculating the vanishing point coordinates, and then using these coordinates. Figure 4 In the image plane, the coordinates of the vanishing point are (dx, dy). Therefore, the pitch and roll transformation angles ω and β from the three-dimensional spatial coordinate system to the camera coordinate system can be calculated, ignoring orientation:

[0062]

[0063]

[0064] S3.2 Vertical Vanishing Point Calculation

[0065] In the set of vertical line segments obtained in step 2, the intersection coordinates of any two line segments can be calculated, which are the observations of the vanishing points in the vertical direction of the image. The longer the extracted line segment, the more accurate the calculation of the intersection coordinates; shorter line segments have larger errors in the intersection coordinate calculations. Therefore, the observation accuracy of each vanishing point observation varies. Using the product of the lengths of two line segments as the weight of the observation points, a weighted distribution of vanishing point observations is obtained, such as... Figure 5 As shown.

[0066] S3.3 Vertical Vanishing Point Cluster Center Calculation

[0067] DBSCAN is used to cluster high-weight vanishing points, and the coordinates of the cluster centers are calculated to obtain optimized vanishing point measurement results, such as... Figure 6 As shown.

[0068] S4. Gravity Direction Calculation and Restoration

[0069] Substituting the vanishing point location calculation result into the formula in step 3.1, the camera pitch and roll angles are calculated. The image is then reconstructed using these angles, as shown in the following figure. Figure 7 As shown in the figure, all vertical line segments have been restored to their vertical positions, thus proving that this method can complete the calculation of the camera's gravity direction.

Claims

1. A camera gravity direction self-correction method based on vanishing point, characterized in that, Using vehicle driving record data, by extracting and calculating vertical line segments in the image, and based on the single-aperture camera imaging model, the roll angle and pitch angle of the image are calculated, and the camera's gravity direction attitude is restored in the absence of gravity data. Includes the following steps: S1. Extraction of features of line segments in the image; Based on the input image, line segments in the image are identified through line detection, and vertical line segments are selected. Image line segment features are implemented using the LSD fast line detection algorithm; S2. Extraction of vertical line segment features from the image By setting a threshold for the rotation angle of line segments relative to the y-direction of the image, a batch of line segments that meet the slope threshold are selected from the line segment extraction results. At the same time, there are many fragmented line segments in the extraction results, so the length of the line segments also needs to be selected. The final selection result is used as the spatial vertical line segment result for calculating the vanishing point position. S3, Calculation of image gravity direction S3.1 Image Gravity Direction Calculation Method Suppose there are two vertical line segments AB and CD in two segments of space, and the spatial coordinates of four points ABCD are as follows: , , and Let P be the intersection point of two spatial vertical lines in the image plane, and its coordinates be... ; Let the camera's extrinsic parameters be... and Where R represents the rotation matrix between the camera coordinate system and the spatial coordinate system. This represents the translation matrix between the two coordinate systems. Since the current calculation only involves rotation, therefore... ,set up The components are as follows: , Coordinates of a point in space Image point coordinates on the photograph They satisfy the collinear relationship, where Here are the camera's coordinates in the spatial coordinate system: , Substituting the coordinates of the four points, we obtain the equations of the lines projected onto the image plane by the two spatial lines: Line AB: , Line CD: , The intersection of AB and CD is calculated. Coordinates are This coordinate is independent of the spatial point coordinates, and only depends on the camera's spatial rotation attitude; therefore, when the image plane has a certain angle with the vertical direction, any vertical line in space will intersect at a point in the image plane, and this point is the vanishing point of this set of parallel lines in the current image; because Camera coordinate system With spatial three-dimensional coordinate system The conversion relationship between them is as follows: , When only considering the camera's pitch and roll changes, i.e., the camera first rotates... Axis rotation That is, pitch change, then rotation Axis rotation That is, by performing a roll transformation, the image plane coordinates of the vanishing point are obtained. Its coordinates in the three-dimensional coordinate system are That is, the intersection of the vertical line passing through the center of the camera projection and the image plane in the three-dimensional coordinate system; By extracting the contours of vertical features, calculating the coordinates of the vanishing point, and then using these coordinates to calculate the roll and pitch angles; the coordinates of the vanishing point in the image plane are... Calculate the pitch and roll transformation angles from the spatial 3D coordinate system to the camera coordinate system, ignoring orientation. and : , , S3.2 Calculation of Vertical Vanishing Point In the set of vertical line segments obtained in step S2, the coordinates of the intersection point between any two line segments can be calculated, which is the observation value of the vanishing point in the vertical direction of the image; the product of the lengths of the two line segments is used as the weight of the observation point to obtain the weighted distribution of the vanishing point observation value. S3.3 Calculation of Cluster Centers at Vertical Vanishing Points DBSCAN is used to cluster high-weight vanishing points and the coordinates of the cluster centers are calculated to obtain optimized vanishing point measurement results. S4. Gravity Direction Calculation and Restoration Substitute the result of the vanishing point position calculation into the formula in step S3.1 to calculate the camera pitch angle and roll angle, and use these angles to recover the image.

Images