External parameter calibration method and device based on vehicle-mounted camera

Abstract

The invention provides an external parameter calibration method and device based on a vehicle-mounted camera, and the method comprises the steps: obtaining a motion parameter set collected by a speedometer in real time, marking a target frame motion parameter and a target frame timestamp from the motion parameter set, and marking a reference frame motion parameter and a reference frame timestamp from the motion parameter set; obtaining a target image according to the target frame timestamp, and obtaining an image coordinate of a target feature point in the target image; generating an external parameter matrix according to the image coordinates of the target feature points, and calculating a jolting angle according to the target frame motion parameters and the reference frame motion parameters; and generating a compensation matrix based on the jolting angle, and dividing the external parameter matrix by the compensation matrix to calculate an external parameter calibration matrix. According to the method, the bump angle between the target frame and the reference frame is calculated to perform motion compensation on the calibration result of the vehicle-mounted camera caused by bump, so that the calibration accuracy of the vehicle-mounted camera is improved.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to an external parameter calibration method and device based on a vehicle-mounted camera. Background Art

[0002] The on-board camera is a core sensor used to perceive the vehicle environment. It can provide real-time road information, obstacle locations, traffic signs and other data information, and detect whether the vehicle deviates from the lane based on this data information, thereby improving driving safety. In order to ensure the accuracy of the data provided by the on-board camera, it is necessary to calibrate the extrinsic parameter matrix of the on-board camera. The extrinsic parameter matrix is ​​used to describe the position and posture of the on-board camera in the world coordinate system. Based on the extrinsic parameter matrix, the position and direction of the image data collected by the on-board camera in three-dimensional space can be calculated.

[0003] The extrinsic matrix can be calibrated by placing a fixed calibration plate with a specific pattern in the static calibration scene and using the vehicle camera to take multiple sets of images. However, during the driving process, the extrinsic matrix of the vehicle camera will be offset due to various factors, such as vehicle bumps caused by uneven road surface, abnormal lane markings, etc. These abnormalities will cause the calibration results of the extrinsic matrix to deviate greatly from the actual situation, thereby reducing the accuracy and reliability of the vehicle camera. Summary of the invention

[0004] The present application provides an extrinsic parameter calibration method and device based on a vehicle-mounted camera, which performs motion compensation on the calibration result of the vehicle-mounted camera caused by bumps, thereby improving the accuracy of the vehicle-mounted camera calibration.

[0005] In a first aspect, the present application provides an external parameter calibration method based on a vehicle-mounted camera, comprising:

[0006] Acquire a motion parameter set collected in real time by the odometer, the motion parameter set including motion parameters and motion timestamps, the motion parameters and the motion timestamps being associated with each other;

[0007] Marking target frame motion parameters and target frame timestamps from the motion parameter set, and marking reference frame motion parameters and reference frame timestamps from the motion parameter set, the reference frame timestamp being less than the target frame timestamp;

[0008] Acquire a target image according to the target frame timestamp, and acquire image coordinates of target feature points in the target image, wherein the target image is image data acquired by a vehicle-mounted camera at the target frame timestamp, and the target image includes at least two lane lines;

[0009] Generate an extrinsic parameter matrix according to the image coordinates of the target feature points, wherein the extrinsic parameter matrix is ​​used to represent the rotation angle of the camera coordinate system of the vehicle-mounted camera relative to the world coordinate system;

[0010] Calculating a bump angle according to the target frame motion parameters and the reference frame motion parameters, the bump angle being used to characterize a rotation angle of the vehicle-mounted camera within a calibration time period, the calibration time period being a difference between a timestamp of the target frame and a timestamp of the reference frame;

[0011] A compensation matrix is ​​generated based on the bump angle, and the extrinsic parameter matrix is ​​divided by the compensation matrix to calculate an extrinsic parameter calibration matrix.

[0012] In some implementations, marking the target frame motion parameter from the motion parameter set includes:

[0013] Calculating the driving speed according to the acceleration, and establishing an association relationship between the driving speed, the motion timestamp and the motion parameter;

[0014] If the driving speed is greater than or equal to a preset speed threshold, obtaining the motion parameter associated with the driving speed, and marking the motion parameter as a target frame motion parameter;

[0015] If the driving speed is less than a preset speed threshold, the motion parameter associated with the driving speed is obtained, and the motion parameter is marked as a non-target frame motion parameter.

[0016] In some implementations, marking the target frame motion parameter from the motion parameter set includes:

[0017] Calculating the driving speed according to the acceleration, and establishing an association relationship between the driving speed, the motion timestamp and the motion parameter;

[0018] If the driving speed is greater than or equal to the preset speed threshold, a first image captured by the vehicle-mounted camera is acquired, where the acquisition time of the first image is a motion timestamp corresponding to the driving speed;

[0019] Converting the first image into a second image using an image processing model, wherein the second image is a bird's-eye view of the first image, and the second image includes lane lines;

[0020] Establishing a two-dimensional coordinate system of the second image, and calculating a straight line expression of the lane line in the second image, wherein the straight line expression includes a slope parameter and an intercept parameter;

[0021] If the absolute value of the slope parameter is less than or equal to a preset slope threshold, and the intercept parameter is within a preset intercept interval, then obtaining the motion parameter associated with the driving speed, and marking the motion parameter as a target frame motion parameter;

[0022] If the absolute value of the slope parameter is greater than a preset slope threshold, or the intercept parameter is not within a preset intercept interval, the motion parameter associated with the driving speed is obtained, and the motion parameter is marked as a non-target frame motion parameter.

[0023] In some implementations, the extrinsic parameter calibration method based on a vehicle-mounted camera further includes:

[0024] If the driving speed is less than a preset speed threshold, the motion parameter associated with the driving speed is obtained, and the motion parameter is marked as a non-target frame motion parameter.

[0025] In some implementations, obtaining the image coordinates of the target feature points in the target image includes:

[0026] Establishing a two-dimensional coordinate system of the target image;

[0027] Identifying lane lines in the target image using a lane line detection algorithm;

[0028] Extending the lane line to obtain the target feature point, where the target feature point is the intersection point of the extended lane line;

[0029] The image coordinates of the target feature point are obtained, where the image coordinates are coordinate values ​​of the target feature point in the two-dimensional coordinate system.

[0030] In some implementations, calculating the turbulence angle according to the target frame motion parameters and the reference frame motion parameters includes:

[0031] Extracting the angular velocity within the calibration time period from the motion parameter set, the angular velocity comprising a first axis angular velocity, a second axis angular velocity and a third axis angular velocity;

[0032] performing an integration operation on the first axis angular velocity within the calibration time period to calculate a yaw angle;

[0033] Performing an integration operation on the angular velocity of the second axis within the calibration time period to calculate a top-down angle;

[0034] An integration operation is performed on the third axis angular velocity within the calibration period to calculate a roll angle.

[0035] In some implementations, generating a compensation matrix based on the pitch angle includes:

[0036] generating a yaw rotation matrix according to the yaw angle;

[0037] Generate a top-view rotation matrix according to the top-view angle;

[0038] The roll angle is set to a preset angle, and a roll rotation matrix is ​​generated according to the preset angle;

[0039] The yaw rotation matrix, the pitch rotation matrix, and the roll rotation matrix are multiplied to calculate a compensation matrix.

[0040] In some implementations, the extrinsic parameter calibration method based on a vehicle-mounted camera further includes:

[0041] Acquire a reference image according to the reference frame timestamp, and acquire image coordinates of reference feature points in the reference image, wherein the reference image is image data acquired by a vehicle-mounted camera at the reference frame timestamp, and the reference image includes at least two lane lines;

[0042] Generate a reference extrinsic parameter matrix according to the image coordinates of the reference feature points, and establish a reference coordinate system based on the reference extrinsic parameter matrix;

[0043] Projecting the reference image into the reference coordinate system to obtain lane line reference coordinates;

[0044] Establishing a calibration coordinate system based on the extrinsic calibration matrix, and projecting the target image into the calibration coordinate system to obtain lane line calibration coordinates;

[0045] Calculating the error discreteness between the lane line reference coordinates and the lane line calibration coordinates;

[0046] The calibration state of the extrinsic parameter calibration matrix is ​​marked according to the error dispersion, and the calibration state includes a success state and a failure state.

[0047] In some implementations, marking the calibration state of the extrinsic calibration matrix according to the error dispersion includes:

[0048] If the error dispersion is less than or equal to the preset dispersion, marking the calibration state of the extrinsic parameter calibration matrix as a success state;

[0049] If the error discreteness is greater than a preset discreteness, the calibration state of the extrinsic parameter calibration matrix is ​​marked as a failure state.

[0050] In a second aspect, an embodiment of the present application proposes an external parameter calibration device based on a vehicle-mounted camera, comprising:

[0051] An information collection module is configured to obtain a set of motion parameters collected by the odometer in real time, wherein the set of motion parameters includes motion parameters and motion timestamps, and the motion parameters are associated with the motion timestamps;

[0052] a parameter extraction module configured to mark a target frame motion parameter and a target frame timestamp from the motion parameter set, and mark a reference frame motion parameter and a reference frame timestamp from the motion parameter set, the reference frame timestamp being less than the target frame timestamp;

[0053] an image processing module, configured to acquire a target image according to the target frame timestamp, and acquire image coordinates of target feature points in the target image, wherein the target image is image data acquired by a vehicle-mounted camera at the target frame timestamp, and the target image includes at least two lane lines;

[0054] A matrix generation module is configured to generate an extrinsic parameter matrix according to the image coordinates of the target feature point, wherein the extrinsic parameter matrix is ​​used to represent the rotation angle of the camera coordinate system of the vehicle-mounted camera relative to the world coordinate system;

[0055] a data processing module configured to calculate a pitch angle according to the target frame motion parameters and the reference frame motion parameters, wherein the pitch angle is used to characterize the rotation angle of the vehicle-mounted camera within a calibration time period, wherein the calibration time period is a difference between the target frame timestamp and the reference frame timestamp;

[0056] The matrix generation module is further configured to generate a compensation matrix based on the bump angle, and divide the extrinsic parameter matrix by the compensation matrix to calculate an extrinsic parameter calibration matrix.

[0057] The present application provides an extrinsic parameter calibration method and device based on a vehicle-mounted camera, the method comprising: obtaining a set of motion parameters collected in real time by an odometer; marking the target frame motion parameters and the target frame timestamp from the motion parameter set, marking the reference frame motion parameters and the reference frame timestamp from the motion parameter set; obtaining a target image according to the target frame timestamp, and obtaining the image coordinates of the target feature points in the target image; generating an extrinsic parameter matrix according to the image coordinates of the target feature points, and calculating the bump angle according to the target frame motion parameters and the reference frame motion parameters; generating a compensation matrix based on the bump angle, and dividing the extrinsic parameter matrix by the compensation matrix to calculate the extrinsic parameter calibration matrix. The above method performs motion compensation on the vehicle-mounted camera calibration result caused by bumps by calculating the bump angle between the target frame and the reference frame, thereby improving the accuracy of the vehicle-mounted camera calibration. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the embodiments are briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0059] Figure 1 A schematic diagram of the installation position of a vehicle-mounted camera according to an embodiment of the present application;

[0060] Figure 2 A schematic diagram of the steps of an extrinsic parameter calibration method based on a vehicle-mounted camera according to an embodiment of the present application;

[0061] Figure 3 A schematic diagram of the steps of screening motion parameters according to driving speed in an embodiment of the present application;

[0062] Figure 4 A schematic diagram of the steps of selecting motion parameters according to the curvature of the lane line in an embodiment of the present application;

[0063] Figure 5 A schematic diagram of the steps of obtaining the image coordinates of the target feature points according to an embodiment of the present application;

[0064] Figure 6 A schematic diagram of the steps of calculating the compensation matrix according to an embodiment of the present application;

[0065] Figure 7 A schematic diagram of the steps of marking the calibration state of the extrinsic parameter calibration matrix in an embodiment of the present application;

[0066] Figure 8 Schematic diagram of the structure of an external parameter calibration device based on a vehicle-mounted camera according to an embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present application.

[0068] It should be understood that the "multiple" mentioned herein refers to two or more. In the description of the embodiments of the present application, unless otherwise specified, " / " means or, for example, A / B can mean A or B; "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, in order to facilitate the clear description of the technical solution of the embodiments of the present application, in the embodiments of the present application, the words "first", "second" and the like are used to distinguish between the same items or similar items with basically the same functions and effects. Those skilled in the art can understand that the words "first", "second" and the like do not limit the quantity and execution order, and the words "first", "second" and the like do not limit them to be necessarily different.

[0069] In addition, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus that includes a series of steps or elements is not necessarily limited to those steps or elements explicitly listed, but may include other steps or elements not explicitly listed or inherent to such process, method, product, or apparatus.

[0070] The on-board camera is a core sensor used to perceive the vehicle environment. It can provide real-time road information, obstacle locations, traffic signs and other data information, and detect whether the vehicle deviates from the lane based on this data information, thereby improving driving safety. In order to ensure the accuracy of the data provided by the on-board camera, it is necessary to calibrate the extrinsic parameter matrix of the on-board camera. The extrinsic parameter matrix is ​​used to describe the position and attitude parameters of the on-board camera in the world coordinate system. Based on the extrinsic parameter matrix, the position and direction of the image data collected by the on-board camera in three-dimensional space can be calculated.

[0071] like Figure 1 As shown, the vehicle body coordinate system is the center position of the rear wheel axle in the figure, wherein the Y-axis direction is the same as the wheel axle direction, the X-axis direction is perpendicular to the Y-axis and points to the forward direction of the longitudinal axis, and the Z-axis direction is the upward direction based on the right-hand rule. The vehicle-mounted camera is fixedly installed on the vehicle body. For example, the front wide-angle camera installed on the front side of the vehicle body has a larger field of view and can see the lane line in front. The cameras installed on the left and right sides of the vehicle body can observe part of the lane line on one side.

[0072] During the driving process of the vehicle, the external parameter matrix data will be affected by many factors and errors will occur, such as: vehicle vibration, temperature changes, and wear and tear of vehicle parts, which will affect the accuracy of the camera data. Therefore, it is necessary to calibrate the external parameter matrix of the vehicle camera. The goal of the external parameter matrix calibration of the vehicle camera is to calibrate the camera coordinate system to the vehicle coordinate system.

[0073] The extrinsic matrix can be calibrated by placing a fixed calibration plate with a specific pattern in the static calibration scene and using the vehicle camera to take multiple sets of images. However, during the driving process, the extrinsic matrix of the vehicle camera will be offset due to various factors, such as vehicle bumps caused by uneven road surface, abnormal lane markings, etc. These abnormalities will cause the calibration results of the extrinsic matrix to deviate greatly from the actual situation, thereby reducing the accuracy and reliability of the vehicle camera.

[0074] In order to perform motion compensation on the calibration result of the vehicle-mounted camera and improve the accuracy of the vehicle-mounted camera calibration, an embodiment of the present application provides an extrinsic parameter calibration method based on the vehicle-mounted camera. Figure 2 This is a schematic diagram of the steps of the external parameter calibration method based on the vehicle-mounted camera in the embodiment of the present application. Figure 2 The method of this embodiment is described in detail.

[0075] Step S100: Obtain a set of motion parameters collected by the odometer in real time.

[0076] It should be understood that the autonomous vehicle obtains its own vehicle motion information through the odometer, and the self-driving vehicle motion information includes but is not limited to information such as speed, acceleration, yaw angular velocity, position, direction and attitude. In some embodiments, the autonomous vehicle can also obtain its own vehicle motion information through an inertial measurement unit (IMU). An inertial measurement unit is a device that integrates multiple inertial sensors, including accelerometers, gyroscopes, and magnetometers, etc., and is used to measure and track the vehicle's motion state.

[0077] In some embodiments, the motion parameter set includes motion parameters and motion timestamps, and the motion parameters are associated with the motion timestamp, that is, each motion timestamp corresponds to a set of motion parameters, and the motion parameters include but are not limited to vehicle angular velocity and acceleration. For example: the motion timestamp is 13:20:11 on January 1, 2023, the vehicle angular velocity is 0.6 radians per second, and the acceleration is 2.3 meters per second squared.

[0078] Step S200: Mark the target frame motion parameters and the target frame timestamp from the motion parameter set, and mark the reference frame motion parameters and the reference frame timestamp from the motion parameter set.

[0079] In order to improve the calibration efficiency of the extrinsic parameter matrix of the vehicle-mounted camera, it is necessary to screen the data in the motion parameter set to ensure the validity of the calibration data. In some embodiments, the target frame motion parameters and the target frame timestamp are marked from the motion parameter set, and the reference frame motion parameters and the reference frame timestamp are marked. Among them, the reference frame timestamp is less than the target frame timestamp. The reference frame motion parameters can be used as reference motion data. By comparing the difference between the target frame motion parameters and the reference frame motion parameters, the vehicle's bump angle can be calculated, and the vehicle-mounted camera calibration result can be motion compensated based on the bump angle, thereby improving the accuracy of the vehicle-mounted camera calibration.

[0080] In some embodiments, due to the low speed of the vehicle, the vehicle equipment will collect too much data at the same scene position, resulting in a large amount of invalid data in the motion parameter set, causing local convergence of the calibration results, thereby reducing the accuracy of the calibration results. Figure 3As shown, the driving speed is calculated according to the acceleration, and the association between the driving speed, the motion timestamp and the motion parameters is established to filter the motion parameters according to the driving speed. If the driving speed is greater than or equal to the preset speed threshold, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as target frame motion parameters; if the driving speed is less than the preset speed threshold, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as non-target frame motion parameters.

[0081] For example, the motion timestamp is 10:20:11 on January 2, 2023, the vehicle angular velocity is 2.5 radians per second, and the acceleration is 3 meters per second squared. By performing an integral operation on the angular velocity and acceleration, the driving speed can be calculated to be 5.3 meters per second. The preset vehicle speed threshold is 0.3 meters per second. If the driving speed is greater than the preset vehicle speed threshold, the group of motion parameters is marked as the target frame motion parameters.

[0082] For example, the motion timestamp is 10:20:11 on January 2, 2023, the vehicle angular velocity is 0.91 radians per second, and the acceleration is 0.12 meters per second squared. By performing integral operations on the angular velocity and acceleration, the vehicle speed can be calculated to be 0.22 meters per second. The preset vehicle speed threshold is 0.3 meters per second. If the vehicle speed is less than the preset vehicle speed threshold and the vehicle tends to be stationary, the group of motion parameters is marked as non-target frame motion parameters, thereby reducing the accumulation of vehicle stationary data.

[0083] Through the above-mentioned preset motion screening strategy, the motion parameters with a vehicle speed greater than or equal to the preset vehicle speed threshold are marked from the motion parameter set as the target frame motion parameters, which can reduce the accumulation of vehicle static data and improve the efficiency and accuracy of calibration.

[0084] In some embodiments, the preset motion screening strategy also includes detecting the curvature of lane lines in the image captured by the vehicle-mounted camera. Figure 4 As shown, the driving speed is first calculated according to the acceleration, and the correlation between the driving speed, the motion timestamp and the motion parameters is established. Then the driving speed is compared with the value of the preset speed threshold. If the driving speed is less than the preset speed threshold, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as non-target frame motion parameters. Based on this, the data with a driving speed less than the preset speed threshold is removed from the motion parameter set to reduce the accumulation of vehicle static data. If the driving speed is greater than or equal to the preset speed threshold, the first image captured by the vehicle-mounted camera is obtained, and the acquisition time of the first image is the motion timestamp corresponding to the driving speed.

[0085] It should be understood that the on-board camera collects image information of the vehicle's surrounding environment in real time to provide the autonomous driving system with real-time environmental perception capabilities. The image information is associated with the collection time. For example, the first image is image data collected by the on-board camera at the first time, and the first image includes at least two lane lines.

[0086] Lane lines are important signs on the road surface that indicate the vehicle's route. The on-board camera collects lane line image data to provide navigation information for the autonomous driving system to prevent the vehicle from deviating from the intended driving path and reduce the number of vehicles entering prohibited areas or violating traffic regulations. By analyzing and processing lane line image data, the algorithm model of the autonomous driving system can be optimized to improve driving safety.

[0087] Furthermore, the image processing model is used to convert the first image into a second image, where the second image is a bird's-eye view of the first image, and the second image includes lane lines. In some embodiments, the image processing model first performs denoising and correction on the first image to eliminate noise interference in the first image and eliminate image deformation caused by lens distortion. Then, a perspective transformation is performed on the first image to map points in the first image plane to the second image plane, and then multi-view geometry and image fusion technology are used to generate a bird's-eye view of the first image, i.e., the second image.

[0088] Furthermore, a two-dimensional coordinate system of the second image is established, and a straight line expression of the lane line in the second image is calculated, wherein the straight line expression includes a slope parameter and an intercept parameter. Next, based on the straight line expression of the lane line in the second image, the target frame motion parameters are marked. If the absolute value of the slope parameter is less than or equal to the preset slope threshold, and the intercept parameter is within the preset intercept interval, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as target frame motion parameters; if the absolute value of the slope parameter is greater than the preset slope threshold, or the intercept parameter is not within the preset intercept interval, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as non-target frame motion parameters.

[0089] It should be noted that since the lane line of the turning part is not a straight line in the bird's-eye view, it is easy to have lane line recognition errors, so it is necessary to remove the turning data in the motion parameter set to improve the calibration accuracy of the vehicle-mounted camera. By establishing a two-dimensional coordinate system for the second image and calculating the straight line expression of the lane line in the second image, the straightness and position of the lane line can be judged.

[0090] In some embodiments, the lane line is represented by the following straight line equation:

[0091] y=kx+b

[0092] Wherein, k represents the slope parameter, b represents the intercept parameter, x represents the X-axis coordinate value, and y represents the Y-axis coordinate value. When the absolute value of the slope parameter is less than or equal to the preset slope threshold, and the intercept parameter is within the preset intercept interval, it indicates that the lane line is a straight line, and the position of the lane line conforms to the preset distribution interval. At this time, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as target frame motion parameters. When the absolute value of the slope parameter is greater than the preset slope threshold, or the intercept parameter is not within the preset intercept interval, it indicates that the lane line is not a straight line, or the position of the lane line does not conform to the preset distribution interval. At this time, the motion parameters associated with the driving speed are obtained, and the motion parameters are marked as non-target frame motion parameters.

[0093] Through the above-mentioned preset motion screening strategy, the target frame motion parameters and non-target frame motion parameters are marked from the motion parameter set, and the external parameter matrix of the vehicle-mounted camera is calibrated based on the target frame motion parameters. This can reduce the accumulation of vehicle static data and the accumulation of turning motion data, thereby improving the efficiency and accuracy of calibration.

[0094] Step S300: acquiring a target image according to a target frame timestamp, and acquiring image coordinates of target feature points in the target image.

[0095] In some embodiments, the target image is image data collected by the vehicle-mounted camera at the target frame timestamp, and the target image includes at least two lane lines. The image coordinates of the target feature points in the target image can be obtained by using the lane line detection algorithm, such as Figure 5 As shown in the figure, first establish a two-dimensional coordinate system of the target image; then use the lane line detection algorithm to identify the lane lines in the target image; then extend the lane lines to obtain the target feature points and the image coordinates of the target feature points. Among them, the target feature points are the intersection points after the lane lines are extended, and the image coordinates are the coordinate values ​​of the target feature points in the two-dimensional coordinate system.

[0096] It should be noted that the lane lines are two or more parallel lines. In a two-dimensional image, the lane lines extend far away and intersect at a point, which is called a vanishing point (VP). By detecting the vanishing point in the image, the direction of the road and the position information of the vehicle can be inferred. In an embodiment of the present application, the target feature point can be a vanishing point. By establishing a two-dimensional coordinate system of the target image and extending the lane lines in the target image, the vanishing point and the two-dimensional coordinates of the vanishing point can be obtained.

[0097] Step S400: Generate an extrinsic parameter matrix according to the image coordinates of the target feature points.

[0098] The extrinsic matrix is ​​used to characterize the rotation angle of the camera coordinate system of the vehicle-mounted camera relative to the world coordinate system. Through the extrinsic matrix, the conversion of points in the camera coordinate system to the world coordinate system can be realized. In some embodiments, the target feature point contains the rotation posture information of the vehicle-mounted camera, so the extrinsic matrix of the vehicle-mounted camera can be calculated according to the image coordinates of the target feature point.

[0099] Exemplarily, the intrinsic parameter matrix of the camera is obtained during the camera calibration process, wherein the intrinsic parameter matrix includes internal parameters of the camera such as focal length and optical center. Since the image coordinates of the target feature points are the projection of the extrinsic parameter matrix on the image plane after being transformed by the intrinsic parameter matrix, the extrinsic parameter matrix can be calculated using the intrinsic parameter matrix and the image coordinates of the target feature points.

[0100] Step S500: Calculate the bump angle according to the target frame motion parameters and the reference frame motion parameters.

[0101] During vehicle driving, uneven road surface will cause the vehicle to bump, which will cause the extrinsic matrix of the vehicle camera to shift, thus affecting the calibration effect of the extrinsic matrix. Therefore, it is necessary to calculate the bump angle of the vehicle and perform motion compensation on the calibration result of the vehicle camera caused by the bump, thereby improving the accuracy of the vehicle camera calibration.

[0102] In some embodiments, the bump angle is used to characterize the rotation angle of the vehicle-mounted camera within a calibration period, and the calibration period is the difference between the target frame timestamp and the reference frame timestamp. The vehicle-mounted camera generates a bump angle due to the uneven road surface between the reference frame timestamp and the target frame timestamp. The bump angle can be obtained by extracting the angular velocity within the calibration period from the motion parameter set and performing an integral operation on the angular velocity.

[0103] In some embodiments, the angular velocity includes a first-axis angular velocity, a second-axis angular velocity, and a third-axis angular velocity. The yaw angle is calculated by performing an integral operation on the first-axis angular velocity within a calibration time period; and the look-down angle is calculated by performing an integral operation on the second-axis angular velocity within a calibration time period; and the roll angle is calculated by performing an integral operation on the third-axis angular velocity within the calibration time period. The yaw angle, the look-down angle, and the roll angle together constitute the pitch angle.

[0104] Step S600: Generate a compensation matrix based on the bump angle, and divide the extrinsic parameter matrix by the compensation matrix to calculate an extrinsic parameter calibration matrix.

[0105] In some embodiments, the pitch angle is used to describe the rotation angle of the vehicle-mounted camera in three-dimensional space, such as Figure 6As shown, a yaw rotation matrix is ​​generated according to the yaw angle; and a top-view rotation matrix is ​​generated according to the top-view angle; and the roll angle is set to a preset angle (e.g., 0 degrees), and a roll rotation matrix is ​​generated according to the preset angle. The yaw rotation matrix, the top-view rotation matrix, and the roll rotation matrix are then multiplied to calculate a compensation matrix. The extrinsic parameter matrix is ​​divided by the compensation matrix to calculate an extrinsic parameter calibration matrix. The extrinsic parameter calibration matrix is ​​used to calibrate the vehicle-mounted camera, which can reduce the impact of vehicle bumps on the calibration effect, thereby improving the calibration accuracy.

[0106] In some embodiments, in order to verify the validity of the external parameter calibration matrix, it is necessary to mark the state of the external parameter calibration matrix. Figure 7 As shown, a reference image is obtained according to the reference frame timestamp, and the image coordinates of the reference feature points in the reference image are obtained. The reference image is the image data collected by the vehicle-mounted camera at the reference frame timestamp, and the reference image includes at least two lane lines. The lane lines in the reference image are used as the standard to verify the calibrated lane line positions, and then the accuracy of the external parameter calibration matrix is ​​determined.

[0107] A reference extrinsic matrix is ​​generated according to the image coordinates of the reference feature points, and a reference coordinate system is established based on the reference extrinsic matrix; the reference image is then projected into the reference coordinate system to obtain the lane line reference coordinates. Next, a calibration coordinate system is established based on the extrinsic calibration matrix, and the target image is projected into the calibration coordinate system to obtain the lane line calibration coordinates. The accuracy of the extrinsic calibration matrix can be obtained by calculating the error discreteness between the lane line reference coordinates and the lane line calibration coordinates. For example, the smaller the error discreteness between the lane line calibration coordinates and the lane line reference coordinates, the higher the overlap between the lane line in the target image and the lane line in the reference image.

[0108] In some embodiments, if the error discreteness is less than or equal to the preset discreteness, the calibration state of the extrinsic calibration matrix is ​​marked as a successful state; if the error discreteness is greater than the preset discreteness, the calibration state of the extrinsic calibration matrix is ​​marked as a failed state. If the calibration state of the extrinsic calibration matrix is ​​a successful state, the calibration result is accepted and the extrinsic parameter matrix of the vehicle-mounted camera is updated; if the calibration state of the extrinsic calibration matrix is ​​a failed state, the calibration result is rejected and the extrinsic calibration matrix corresponding to the motion parameters of the next target frame is calculated.

[0109] It can be seen from the above technical solutions that the present application provides an extrinsic parameter calibration method and device based on a vehicle-mounted camera, the method comprising: obtaining a set of motion parameters collected in real time by the odometer, marking the target frame motion parameters and the target frame timestamp from the motion parameter set, marking the reference frame motion parameters and the reference frame timestamp from the motion parameter set; and obtaining the target image according to the target frame timestamp, and obtaining the image coordinates of the target feature points in the target image; and generating an extrinsic parameter matrix according to the image coordinates of the target feature points, and calculating the bump angle according to the target frame motion parameters and the reference frame motion parameters; generating a compensation matrix based on the bump angle, and dividing the extrinsic parameter matrix by the compensation matrix to calculate the extrinsic parameter calibration matrix. The above method performs motion compensation on the vehicle-mounted camera calibration result caused by bumps by calculating the bump angle between the target frame and the reference frame, thereby improving the accuracy of the vehicle-mounted camera calibration.

[0110] The method embodiments according to the present application are illustrated above. Based on the extrinsic parameter calibration method based on a vehicle-mounted camera provided in the embodiments of the present application, the embodiments of the present application provide an extrinsic parameter calibration device based on a vehicle-mounted camera. Figure 8 This is a schematic diagram of the structure of the external parameter calibration device based on the vehicle-mounted camera provided in the embodiment of the present application. Figure 8 , the device includes the following modules.

[0111] The information collection module 801 is configured to obtain a set of motion parameters collected by the odometer in real time. The motion parameter set includes motion parameters and motion timestamps. The motion parameters and the motion timestamps are associated with each other.

[0112] The parameter extraction module 802 is configured to mark the target frame motion parameters and the target frame timestamp from the motion parameter set, and mark the reference frame motion parameters and the reference frame timestamp from the motion parameter set, wherein the reference frame timestamp is smaller than the target frame timestamp.

[0113] The image processing module 803 is configured to obtain a target image according to a target frame timestamp and obtain image coordinates of target feature points in the target image. The target image is image data captured by the vehicle-mounted camera at the target frame timestamp. The target image includes at least two lane lines.

[0114] The matrix generation module 804 is configured to generate an extrinsic parameter matrix according to the image coordinates of the target feature points, where the extrinsic parameter matrix is ​​used to represent the rotation angle of the camera coordinate system of the vehicle-mounted camera relative to the world coordinate system.

[0115] The data processing module 805 is configured to calculate the bump angle according to the target frame motion parameters and the reference frame motion parameters. The bump angle is used to characterize the rotation angle of the vehicle-mounted camera within a calibration time period. The calibration time period is the difference between the target frame timestamp and the reference frame timestamp.

[0116] The matrix generation module 804 is further configured to generate a compensation matrix based on the pitch angle, and divide the extrinsic parameter matrix by the compensation matrix to calculate an extrinsic parameter calibration matrix.

[0117] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some or all of the technical features thereof may be replaced by equivalents. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0118] For the convenience of explanation, the above description has been made in conjunction with specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations can be obtained. The selection and description of the above embodiments are to better explain the principles and practical applications, so that those skilled in the art can better use the embodiments and various different variations of the embodiments suitable for specific use considerations.

Claims

1. The external parameter calibration method based on the vehicle-mounted camera is characterized by: include: Acquire a motion parameter set collected in real time by the odometer, the motion parameter set including motion parameters and motion timestamps, the motion parameters and the motion timestamps being associated with each other; Marking target frame motion parameters and target frame timestamps from the motion parameter set, and marking reference frame motion parameters and reference frame timestamps from the motion parameter set, the reference frame timestamp being less than the target frame timestamp; Acquire a target image according to the target frame timestamp, and acquire image coordinates of target feature points in the target image, wherein the target image is image data acquired by a vehicle-mounted camera at the target frame timestamp, and the target image includes at least two lane lines; Generate an extrinsic parameter matrix according to the image coordinates of the target feature points, wherein the extrinsic parameter matrix is ​​used to represent the rotation angle of the camera coordinate system of the vehicle-mounted camera relative to the world coordinate system; Calculating a bump angle according to the target frame motion parameters and the reference frame motion parameters, the bump angle being used to characterize the rotation angle of the vehicle-mounted camera within a calibration time period, the calibration time period being the difference between the target frame timestamp and the reference frame timestamp; A compensation matrix is ​​generated based on the bump angle, and the extrinsic parameter matrix is ​​divided by the compensation matrix to calculate an extrinsic parameter calibration matrix.

2. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 1, characterized in that: The step of marking the target frame motion parameter from the motion parameter set comprises: Calculating the driving speed according to the acceleration, and establishing an association relationship between the driving speed, the motion timestamp and the motion parameter; If the driving speed is greater than or equal to a preset speed threshold, obtaining the motion parameter associated with the driving speed, and marking the motion parameter as a target frame motion parameter; If the driving speed is less than a preset speed threshold, the motion parameter associated with the driving speed is obtained, and the motion parameter is marked as a non-target frame motion parameter.

3. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 1, characterized in that: The step of marking the target frame motion parameter from the motion parameter set comprises: Calculating the driving speed according to the acceleration, and establishing an association relationship between the driving speed, the motion timestamp and the motion parameter; If the driving speed is greater than or equal to the preset speed threshold, a first image captured by the vehicle-mounted camera is acquired, where the acquisition time of the first image is a motion timestamp corresponding to the driving speed; Converting the first image into a second image using an image processing model, wherein the second image is a bird's-eye view of the first image, and the second image includes lane lines; Establishing a two-dimensional coordinate system of the second image, and calculating a straight line expression of the lane line in the second image, wherein the straight line expression includes a slope parameter and an intercept parameter; If the absolute value of the slope parameter is less than or equal to a preset slope threshold, and the intercept parameter is within a preset intercept interval, then obtaining the motion parameter associated with the driving speed, and marking the motion parameter as a target frame motion parameter; If the absolute value of the slope parameter is greater than a preset slope threshold, or the intercept parameter is not within a preset intercept interval, the motion parameter associated with the driving speed is obtained, and the motion parameter is marked as a non-target frame motion parameter.

4. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 3, characterized in that: Also includes: If the driving speed is less than a preset speed threshold, the motion parameter associated with the driving speed is obtained, and the motion parameter is marked as a non-target frame motion parameter.

5. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 1, characterized in that: The obtaining of the image coordinates of the target feature points in the target image comprises: Establishing a two-dimensional coordinate system of the target image; Identifying lane lines in the target image using a lane line detection algorithm; Extending the lane line to obtain the target feature point, where the target feature point is the intersection point of the extended lane line; The image coordinates of the target feature point are obtained, where the image coordinates are coordinate values ​​of the target feature point in the two-dimensional coordinate system.

6. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 1, characterized in that: The calculating the turbulence angle according to the target frame motion parameter and the reference frame motion parameter comprises: Extracting the angular velocity within the calibration time period from the motion parameter set, the angular velocity comprising a first axis angular velocity, a second axis angular velocity and a third axis angular velocity; performing an integration operation on the first axis angular velocity within the calibration time period to calculate a yaw angle; Performing an integration operation on the angular velocity of the second axis within the calibration time period to calculate a top-down angle; An integration operation is performed on the third axis angular velocity within the calibration period to calculate a roll angle.

7. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 6, characterized in that: The generating a compensation matrix based on the pitch angle comprises: generating a yaw rotation matrix according to the yaw angle; Generate a top-view rotation matrix according to the top-view angle; The roll angle is set to a preset angle, and a roll rotation matrix is ​​generated according to the preset angle; The yaw rotation matrix, the pitch rotation matrix, and the roll rotation matrix are multiplied to calculate a compensation matrix.

8. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 1, characterized in that: Also includes: Acquire a reference image according to the reference frame timestamp, and acquire image coordinates of reference feature points in the reference image, wherein the reference image is image data acquired by a vehicle-mounted camera at the reference frame timestamp, and the reference image includes at least two lane lines; Generate a reference extrinsic parameter matrix according to the image coordinates of the reference feature points, and establish a reference coordinate system based on the reference extrinsic parameter matrix; Projecting the reference image into the reference coordinate system to obtain lane line reference coordinates; Establishing a calibration coordinate system based on the extrinsic calibration matrix, and projecting the target image into the calibration coordinate system to obtain lane line calibration coordinates; Calculating the error discreteness between the lane line reference coordinates and the lane line calibration coordinates; The calibration state of the extrinsic parameter calibration matrix is ​​marked according to the error dispersion, and the calibration state includes a success state and a failure state.

9. The extrinsic parameter calibration method based on a vehicle-mounted camera according to claim 8, characterized in that: The step of marking the calibration state of the extrinsic calibration matrix according to the error dispersion comprises: If the error dispersion is less than or equal to the preset dispersion, marking the calibration state of the extrinsic parameter calibration matrix as a success state; If the error discreteness is greater than a preset discreteness, the calibration state of the extrinsic parameter calibration matrix is ​​marked as a failure state.

10. An external parameter calibration device based on a vehicle-mounted camera, characterized in that: The extrinsic parameter calibration method based on a vehicle-mounted camera according to any one of claims 1 to 9, wherein the extrinsic parameter calibration device comprises: An information collection module is configured to obtain a set of motion parameters collected by the odometer in real time, wherein the set of motion parameters includes motion parameters and motion timestamps, and the motion parameters are associated with the motion timestamps; a parameter extraction module configured to mark a target frame motion parameter and a target frame timestamp from the motion parameter set, and mark a reference frame motion parameter and a reference frame timestamp from the motion parameter set, the reference frame timestamp being less than the target frame timestamp; an image processing module, configured to acquire a target image according to the target frame timestamp, and acquire image coordinates of target feature points in the target image, wherein the target image is image data acquired by a vehicle-mounted camera at the target frame timestamp, and the target image includes at least two lane lines; A matrix generation module is configured to generate an extrinsic parameter matrix according to the image coordinates of the target feature point, wherein the extrinsic parameter matrix is ​​used to represent the rotation angle of the camera coordinate system of the vehicle-mounted camera relative to the world coordinate system; a data processing module configured to calculate a pitch angle according to the target frame motion parameters and the reference frame motion parameters, wherein the pitch angle is used to characterize the rotation angle of the vehicle-mounted camera within a calibration time period, wherein the calibration time period is a difference between the target frame timestamp and the reference frame timestamp; The matrix generation module is further configured to generate a compensation matrix based on the bump angle, and divide the extrinsic parameter matrix by the compensation matrix to calculate an extrinsic parameter calibration matrix.

Images