Vehicle-mounted camera external parameter calibration method, device, equipment, medium and program product

Abstract

The invention provides a calibration method and device for external parameters of a vehicle-mounted camera, equipment, a medium and a program product, and relates to the technical field of automatic driving. A vehicle body track of a target vehicle and an image frame collected by a vehicle-mounted camera in the same time period are obtained, the image frame is input into a VO module, key frame information output by the VO module is obtained, and rotation external parameters of the vehicle-mounted camera are calibrated according to the key frame information and the vehicle body track. The method does not depend on a specific use scene and has universality.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to a method, device, equipment, medium and program product for calibrating external parameters of a vehicle-mounted camera. Background Art

[0002] The extrinsic calibration of the camera sensor system refers to determining the rotation and translation relationship between the camera coordinate system and other sensor coordinate systems or the world coordinate system. In the field of autonomous driving, the extrinsic calibration of the camera sensor system is one of the necessary tasks for autonomous driving functions.

[0003] In related technologies, the conversion relationship between the on-board camera and the road surface is calculated based on the road features by detecting road features such as lane lines and road markings, and then the external parameters of the on-board camera relative to the vehicle body are derived by combining the vehicle design parameters such as the installation position of the on-board camera. However, this method has certain limitations. Summary of the invention

[0004] The present application provides a method, device, equipment, medium and program product for calibrating the extrinsic parameters of a vehicle-mounted camera, so as to solve the problem that the calibration method of the extrinsic parameters of a vehicle-mounted camera in the related art has limitations.

[0005] In a first aspect, the present application provides a method for calibrating the extrinsic parameters of a vehicle-mounted camera, comprising: obtaining a vehicle body trajectory of a target vehicle and image frames captured by a vehicle-mounted camera within the same time period; inputting the image frames into a visual odometry (VO) module to obtain key frame information output by the VO module; and calibrating the rotation extrinsic parameters of the vehicle-mounted camera based on the key frame information and the vehicle body trajectory.

[0006] In a possible implementation, the rotational extrinsic parameters of the vehicle-mounted camera are calibrated according to the key frame information and the vehicle trajectory, including: determining the position and posture of the target vehicle according to the key frame information and the vehicle trajectory; and calibrating the rotational extrinsic parameters of the vehicle-mounted camera according to the key frame information, the vehicle trajectory and the position and posture of the target vehicle.

[0007] In a possible implementation, the key frame information includes different key frames and the posture corresponding to each key frame. The posture of the target vehicle is determined according to the key frame information and the vehicle trajectory, including: determining the camera trajectory of the vehicle-mounted camera according to the postures corresponding to two adjacent key frames; for the timestamp corresponding to each image frame in the camera trajectory, interpolating the vehicle trajectory according to the timestamp to obtain the posture of the target vehicle corresponding to the timestamp.

[0008] In a possible implementation, the key frame information also includes a ground plane equation corresponding to the key frame. According to the key frame information, the vehicle trajectory and the position and posture of the target vehicle, the rotation extrinsic parameters of the vehicle-mounted camera are calibrated, including: determining a first residual according to the position and posture corresponding to two adjacent key frames, the position and posture of the target vehicle and the transformation matrix of the rotation extrinsic parameter to be calibrated, the first residual is used to represent the relative displacement between the vehicle-mounted camera and the target vehicle; determining a second residual according to the vehicle trajectory, the camera trajectory, the position and posture of the key frame and the rotation matrix of the rotation extrinsic parameter to be calibrated, the second residual is used to represent the relationship between the vehicle-mounted camera and the forward direction of the target vehicle, and the transformation matrix includes a rotation matrix; determining a third residual according to the ground plane equation, the position and posture of the key frame and the rotation matrix, the third residual is used to represent the relationship between the vehicle-mounted camera and the ground; calibrating the rotation extrinsic parameters of the vehicle-mounted camera according to the first residual, the second residual and the third residual.

[0009] In a possible implementation, a second residual is determined according to a vehicle trajectory, a camera trajectory, a position of a key frame, and a rotation matrix of a rotation extrinsic parameter to be calibrated, including: screening the vehicle trajectory to determine that a trajectory of a uniform straight line in the vehicle trajectory is a target vehicle trajectory; based on a start timestamp and an end timestamp corresponding to the target vehicle trajectory, according to the camera trajectory, determining a 3D point position corresponding to the vehicle-mounted camera; performing spline fitting on the 3D point position to obtain a velocity vector of the vehicle-mounted camera in a world system; and determining a second residual according to the velocity vector and the rotation matrix.

[0010] In one possible implementation, the ground plane equation is obtained in the following manner: obtaining a key frame and a 3D feature in a sliding window, wherein the key frame is marked with road features; for each key frame in the sliding window, determining a preset area according to the position and posture of the key frame; performing plane fitting processing on the 3D features corresponding to the road features in the preset area to obtain the ground plane equation.

[0011] In a possible implementation, the camera trajectory of the vehicle-mounted camera is determined according to the postures corresponding to two adjacent key frames, including: obtaining a fourth residual according to the postures corresponding to two adjacent key frames; obtaining the posture corresponding to the ordinary frame between the two adjacent key frames according to the fourth residual; and determining the camera trajectory according to the order of timestamps corresponding to the key frames and the ordinary frames.

[0012] In a second aspect, the present application provides a device for calibrating extrinsic parameters of a vehicle-mounted camera, comprising:

[0013] An acquisition module is used to acquire the body trajectory of the target vehicle and the image frames captured by the vehicle-mounted camera within the same period of time;

[0014] A processing module is used to input the image frame into the VO module and obtain the key frame information output by the VO module;

[0015] The calibration module is used to calibrate the rotation extrinsic parameters of the vehicle-mounted camera based on the key frame information and the vehicle trajectory.

[0016] In a possible implementation, the calibration module is specifically used to: determine the position and posture of the target vehicle based on the key frame information and the vehicle trajectory; and calibrate the rotation extrinsic parameters of the vehicle-mounted camera based on the key frame information, the vehicle trajectory and the position and posture of the target vehicle.

[0017] In a possible implementation, the key frame information includes different key frames and the posture corresponding to each key frame. The calibration module is also used to: determine the camera trajectory of the vehicle-mounted camera according to the postures corresponding to two adjacent key frames; for the timestamp corresponding to each image frame in the camera trajectory, interpolate the vehicle trajectory according to the timestamp to obtain the posture of the target vehicle corresponding to the timestamp.

[0018] In a possible implementation, the key frame information also includes a ground plane equation corresponding to the key frame, and the calibration module is also used to: determine a first residual based on the postures corresponding to two adjacent key frames, the posture of the target vehicle, and the transformation matrix of the rotation extrinsic parameter to be calibrated, the first residual is used to represent the relative displacement of the vehicle-mounted camera and the target vehicle; determine a second residual based on the vehicle trajectory, the camera trajectory, the posture of the key frame, and the rotation matrix of the rotation extrinsic parameter to be calibrated, the second residual is used to represent the relationship between the vehicle-mounted camera and the forward direction of the target vehicle, and the transformation matrix includes a rotation matrix; determine a third residual based on the ground plane equation, the posture of the key frame, and the rotation matrix, the third residual is used to represent the relationship between the vehicle-mounted camera and the ground; calibrate the rotation extrinsic parameters of the vehicle-mounted camera based on the first residual, the second residual, and the third residual.

[0019] In a possible implementation, the calibration module is also used to: screen the vehicle trajectory to determine that the trajectory of the uniform straight line in the vehicle trajectory is the target vehicle trajectory; based on the start timestamp and end timestamp corresponding to the target vehicle trajectory, according to the camera trajectory, determine the 3D point position corresponding to the vehicle-mounted camera; perform spline fitting on the 3D point position to obtain the velocity vector of the vehicle-mounted camera in the world system; and determine the second residual based on the velocity vector and the rotation matrix.

[0020] In one possible implementation, the ground plane equation is obtained in the following manner: obtaining a key frame and a 3D feature in a sliding window, wherein the key frame is marked with road features; for each key frame in the sliding window, determining a preset area according to the position and posture of the key frame; performing plane fitting processing on the 3D features corresponding to the road features in the preset area to obtain the ground plane equation.

[0021] In a possible implementation, the calibration module is also used to: obtain a fourth residual based on the postures corresponding to two adjacent key frames; obtain a posture corresponding to the ordinary frame between the two adjacent key frames based on the fourth residual; and determine the camera trajectory according to the order of timestamps corresponding to the key frames and the ordinary frames.

[0022] In a third aspect, the present application provides an electronic device comprising at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory is used to store instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method for calibrating the extrinsic parameters of a vehicle-mounted camera provided in the first aspect.

[0023] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are executed by a processor, they are used to implement the method for calibrating the external parameters of a vehicle-mounted camera as provided in the first aspect.

[0024] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program, which, when executed by a processor, implements the method for calibrating the extrinsic parameters of a vehicle-mounted camera as provided in the first aspect.

[0025] The calibration method, device, equipment, medium and program product of the vehicle camera external parameters provided by the present application obtain the vehicle trajectory of the target vehicle and the image frames captured by the vehicle camera in the same period, and input the image frames into the VO module to obtain the key frame information output by the VO module, and calibrate the rotational external parameters of the vehicle camera according to the key frame information and the vehicle trajectory. The present application does not rely on a specific driving trajectory for obtaining the vehicle trajectory and the vehicle camera when capturing image frames, and processes the image frames captured by the vehicle camera through the VO module, and further calibrates the rotational external parameters of the vehicle camera based on the key frame information and the vehicle trajectory output by the VO module, which does not rely on a specific usage scenario and has universal applicability. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0027] Figure 1 Schematic diagram of the process of calibrating the external parameters of the vehicle-mounted camera provided in the embodiment of the present application Figure 1 ;

[0028] Figure 2 A schematic diagram of the structure of the VO module provided in the embodiment of the present application;

[0029] Figure 3Schematic diagram of the process of calibrating the external parameters of the vehicle-mounted camera provided in the embodiment of the present application Figure 2 ;

[0030] Figure 4 The densified VO pose graph provided in the embodiment of the present application;

[0031] Figure 5 Schematic diagram of the process of calibrating the external parameters of the vehicle-mounted camera provided in the embodiment of the present application Figure 3 ;

[0032] Figure 6 A schematic diagram of the structure of a device for calibrating extrinsic parameters of a vehicle-mounted camera provided in one embodiment of the present application;

[0033] Figure 7 A schematic diagram of the structure of an electronic device provided in one embodiment of the present application.

[0034] The above drawings have shown clear embodiments of the present application, which will be described in more detail later. These drawings and text descriptions are not intended to limit the scope of the present application in any way, but to illustrate the concept of the present application to those skilled in the art by referring to specific embodiments. DETAILED DESCRIPTION

[0035] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0036] In the related art, in order to obtain the transformation relationship between the coordinate system of each vehicle-mounted camera and the vehicle body coordinate system, there are usually the following implementation methods: Method 1, using a specific artificial environment, such as a calibration room, target, etc., by shooting the calibration plate or target from multiple angles in the calibration room, obtaining image data from different perspectives, using an image processing algorithm to automatically detect feature points on the calibration plate or target, and calculating the posture of the vehicle-mounted camera based on the position of the feature points, and further calculating the external parameters of the vehicle-mounted camera by minimizing the reprojection error, but this method relies on a specific environment and can usually only be used in the production stage.

[0037] Method 2, based on the hand-eye calibration principle, derives the conversion relationship by comparing the motion trajectories of the vehicle and the camera. That is, by obtaining the motion trajectory of the vehicle and the camera, the motion trajectory of the vehicle and the camera are input into the hand-eye calibration model to obtain the external parameters of the vehicle camera output by the hand-eye calibration model. However, this method requires a specific driving trajectory (such as turning) to constrain each direction, and requires sufficient data volume and high-precision vision and vehicle trajectory to ensure the quality of the results.

[0038] Method 3 uses lane lines and road markings to calculate the conversion relationship between the vehicle camera and the road surface, and then derives the external parameters through the vehicle body design. That is, by detecting road features such as lane lines and road markings, and calculating the conversion relationship between the vehicle camera and the road surface based on the road features, and then combining the vehicle design parameters such as the installation position of the vehicle camera to derive the external parameters of the vehicle camera relative to the vehicle body. However, this method relies on some assumptions, such as clear ground markings, straight and parallel lane lines, and flat ground. These assumptions limit the use scenarios, have certain limitations, and are prone to produce low-precision external parameter results in poor road environments.

[0039] Based on the problems existing in the related technology, the embodiment of the present application obtains the vehicle body trajectory under the free driving trajectory and the image frames captured by the vehicle-mounted camera, and processes the image frames through the VO module to obtain key frame information, and further calibrates the rotational extrinsic parameters of the vehicle-mounted camera based on the key frame information and the vehicle body trajectory. This method does not rely on specific usage scenarios and is universal.

[0040] In order to better understand the calibration method of the vehicle-mounted camera extrinsic parameters provided in the embodiment of the present application, the prerequisites satisfied by the vehicle-mounted camera and the target vehicle are first described:

[0041] 1) The intrinsic parameters of the onboard camera in the target vehicle are known.

[0042] It is understandable that the internal parameters of vehicle-mounted cameras are generally calibrated only once at the factory, and subsequent changes can be ignored.

[0043] 2) The displacement from the vehicle-mounted camera to the vehicle body coordinate system is known.

[0044] It is understandable that the translation accuracy of the vehicle-mounted camera installation and design meets the requirements, so the displacement of the vehicle-mounted camera to the vehicle body coordinate system is known.

[0045] 3) The rotation angle between the vehicle-mounted camera and the vehicle body coordinate system is within a certain design tolerance range.

[0046] Exemplarily, the tolerance range may be ±5°.

[0047] It is understandable that the installation rotation angle of the vehicle-mounted camera within the design tolerance range is a basic condition for the normal operation of each vehicle, and therefore can be used as an initial estimate for the calibration of the vehicle-mounted camera's extrinsic parameters.

[0048] 4) The relative 6-degree-of-freedom (DOF) motion trajectory of the vehicle coordinate system can be obtained.

[0049] The following describes in detail the method for calibrating the extrinsic parameters of a vehicle-mounted camera provided in an embodiment of the present application in conjunction with a specific embodiment.

[0050] Figure 1 Schematic diagram of the process of calibrating the external parameters of the vehicle-mounted camera provided in the embodiment of the present application Figure 1 .

[0051] like Figure 1 As shown, the specific implementation of the calibration method of the vehicle-mounted camera external parameters may include the following steps:

[0052] S101, obtaining the vehicle trajectory of the target vehicle and the image frames captured by the vehicle-mounted camera in the same period of time.

[0053] Exemplarily, the vehicle trajectory may be a vehicle motion trajectory of the target vehicle during driving.

[0054] Exemplarily, the vehicle trajectory can be detected by a dead reckoning (DR) module in the automatic driving system of the target vehicle.

[0055] Exemplarily, the DR module may be composed of a fusion of sensors such as wheel speed, an inertial measurement unit (IMU), and a global navigation satellite system (GNSS).

[0056] Exemplarily, the image frames captured by the vehicle-mounted camera may be a set of image frames within a certain time length.

[0057] It should be noted that in the calibration method of the vehicle-mounted camera extrinsic parameters provided in the embodiment of the present application, there is no limitation on the driving trajectory of the target vehicle for obtaining the vehicle trajectory and the image frame, which can be straight-line driving or turning driving, etc.

[0058] It should be noted that in the calibration method of the vehicle-mounted camera extrinsic parameters provided in the embodiment of the present application, there is no limitation on the specific duration of the same time period corresponding to the vehicle trajectory and the image frame, which can be determined according to actual application requirements.

[0059] S102, input the image frame into the VO module to obtain key frame information output by the VO module.

[0060] Exemplarily, the VO module is a monocular VO module, that is, when the target vehicle is driving, each onboard camera independently runs a VO module.

[0061] Combine the following Figure 2 The VO module provided in the embodiment of the present application is described in detail.

[0062] Figure 2 This is a schematic diagram of the structure of the VO module provided in the embodiment of the present application. Figure 2 As shown, the VO module includes a front end and a back end. The front end is used to extract features of the image frame, match the extracted features, and filter the visual features on the road surface, and the back end is used to perform nonlinear optimization.

[0063] For the front end of the VO module, exemplarily, for any on-board camera in the target vehicle, the image frames collected by the on-board camera are input in real time into the feature extraction unit and road segmentation unit of the front end of the VO module, wherein the feature extraction unit extracts features from the image frames to obtain key features in the image frames and descriptors corresponding to the key features, and further, through the feature matching unit, the key features extracted from adjacent image frames are matched according to the descriptors corresponding to the key features extracted from the adjacent image frames, and then based on the matched features, the current posture of the on-board camera is estimated by the estimation unit; the road segmentation unit performs preliminary screening of the image frames based on the neural network, determines the image frames whose picture changes exceed a certain threshold as key frames, and further marks the road features in the key frames such as road signs, road textures, and road traces, and then compares whether the current posture of the on-board camera and the posture changes of the key frames adjacent to the current posture exceed the threshold, and when the threshold is exceeded, inserts a new key frame, and then uses triangulation to obtain the 3D initial positions of the key features corresponding to all current key frames.

[0064] Exemplarily, the key features may be point features and line features, or point features. In one possible implementation, the feature extraction unit may extract point and line features in the image frame and the descriptors corresponding to the point and line features through a neural network such as the feature network in AirVO; in another possible implementation, the feature extraction unit may extract point features through the ORB algorithm and the SIFT algorithm, and ignore line features.

[0065] Exemplarily, in the feature matching unit, key features may be matched by a neural network (Lightglue).

[0066] Exemplarily, the road segmentation unit may mark the road features in the key frame as 1, that is, mark the pixel points identified as road features in the key frame as 1, and mark the pixel points identified as non-road features as 0; it may also mark the pixel points identified as road features in the key frame as 0, and mark the pixel points identified as non-road features as 1, and so on.

[0067] For the back end of the VO module, exemplarily, in the nonlinear optimization process of the back end, linear optimization is performed on the key frames in the sliding window and the initial positions of the 3D features observed in the sliding window and obtained by triangulation. In a possible implementation, the key frames and 3D initial positions in the sliding window are first selected, and for each key frame in the sliding window, a circular preset area is made with the posture corresponding to the key frame as the center and a preset length as the radius, and the 3D initial position contained in the preset area is determined, and the target 3D initial position marked as a road feature is further selected from the 3D initial position, and the ground plane equation corresponding to the key frame is fitted based on the target 3D initial position, and then the road features and non-road features in the preset area are cleaned to remove erroneous points, and finally the on-board camera posture and 3D initial position are updated by jointly optimizing the reprojection error and the distance error from the on-board camera to the ground plane.

[0068] Exemplarily, the specific implementation method for cleaning 3D road features and non-road features in a preset area can be: on the one hand, for road features and non-road features, follow the general reprojection error to eliminate outliers, that is, project the 3D initial position onto the 2D plane, confirm whether the projection point and the pixel point in the image frame overlap, and when they do not overlap, eliminate the 3D initial position as an outlier; on the other hand, for road features, by checking the distance between the road features and the ground equation, eliminate possible erroneous segmentation marks; secondly, if the number of inner points that conform to the ground equation is less than a certain threshold, that is, the number of road feature points in the preset area is less than a certain threshold or the distance from the road feature to the ground plane is less than a certain threshold, then the ground plane estimation is not performed for this key frame, and the vehicle trajectory output by DR is obtained. If there is obvious angle fluctuation or bump in the vehicle trajectory near the timestamp corresponding to a key frame, the ground plane estimation is considered unreliable, and the ground plane estimation is no longer performed for this key frame.

[0069] Exemplarily, during the iterative optimization process, the 3D initial position and the vehicle camera pose are updated by calculating the change in the 3D initial position and the change in the vehicle camera pose (deltaX) to continue the next iteration. When the residual, i.e., the reprojection error and the error in the distance from the vehicle camera to the ground plane or the above-mentioned change, is less than a certain threshold, the back-end optimization of this time is terminated and the next key frame is waited for.

[0070] It can be understood that in the VO module provided in the embodiment of the present application, during the nonlinear optimization process at the back end, compared with the existing bundle adjustment method (Bundle Adjustment, referred to as BA), the VO module provided in the embodiment of the present application, during the linear optimization process at the back end, in each iteration, will fit the ground plane equation corresponding to the key frame by converging the 3D initial position corresponding to the road surface feature in the preset area of ​​each key frame. Therefore, in the optimization problem, in addition to the traditional reprojection error, the residual also adds the ground plane height error. Based on the fixed displacement and rotation of the on-board camera under the vehicle system to be calibrated, the distance error can be expressed as the distance from the key frame camera pose to the estimated ground plane equation. It should be noted that the residual of the ground plane height error enables the monocular VO scale to be determined, and thus there is no need to specially deal with the scale problem in the subsequent post-processing module.

[0071] It should be noted that in the VO module provided in the embodiment of the present application, there is no requirement for high robustness of the VO module, and it is allowed to restart after failure or loss in some scenarios.

[0072] Exemplarily, when the target vehicle includes multiple on-board cameras, in a possible implementation, when any monocular VO module adds a key frame, all other VO modules add the image frame captured by the on-board camera at that moment as a key frame.

[0073] Exemplarily, the key frame information may include different key frames, and the pose and ground plane equation corresponding to each key frame.

[0074] S103, calibrating the rotation extrinsic parameters of the vehicle-mounted camera according to the key frame information and the vehicle trajectory.

[0075] In one possible implementation, in a nonlinear optimization problem, three residuals are established based on key frame information and vehicle trajectory, namely, the relative displacement between the vehicle-mounted camera and the target vehicle, the relationship between the vehicle-mounted camera and the forward direction of the target vehicle, and the relationship between the vehicle-mounted camera and the ground. The rotation extrinsic parameters of the vehicle-mounted camera are further calibrated based on these three residuals.

[0076] In the embodiment of the present application, the vehicle trajectory of the target vehicle and the image frames captured by the vehicle-mounted camera in the same time period are obtained, and the image frames are input into the VO module to obtain the key frame information output by the VO module, and the rotational extrinsic parameters of the vehicle-mounted camera are calibrated according to the key frame information and the vehicle trajectory. In the embodiment of the present application, the driving trajectory of the vehicle when obtaining the vehicle trajectory and the image frames captured by the vehicle-mounted camera does not depend on a specific driving trajectory, and the image frames captured by the vehicle-mounted camera are processed by the VO module, and the rotational extrinsic parameters of the vehicle-mounted camera are further calibrated based on the key frame information and the vehicle trajectory output by the VO module, which does not depend on a specific usage scenario and has universal applicability.

[0077] Optionally, in a possible implementation method, the specific implementation method of step S103 calibrating the rotational extrinsic parameters of the vehicle-mounted camera according to the key frame information and the vehicle trajectory can be: determining the position and posture of the target vehicle according to the key frame information and the vehicle trajectory; calibrating the rotational extrinsic parameters of the vehicle-mounted camera according to the key frame information, the vehicle trajectory and the position and posture of the target vehicle.

[0078] In a possible implementation, the camera trajectory corresponding to the vehicle-mounted camera and the vehicle trajectory can be obtained according to the posture corresponding to the key frame, and the posture of the target vehicle can be further determined based on the corresponding relationship between the camera trajectory and the vehicle trajectory.

[0079] Optionally, the key frame information includes different key frames and a posture corresponding to each key frame.

[0080] Combine the following Figure 3 The specific implementation method of determining the position and posture of the target vehicle based on the key frame information and the vehicle trajectory is described in detail.

[0081] Figure 3 Schematic diagram of the process of calibrating the external parameters of the vehicle-mounted camera provided in the embodiment of the present application Figure 2 .like Figure 3 As shown, according to the key frame information and the vehicle body trajectory, the specific implementation method of determining the position and posture of the target vehicle may include the following steps:

[0082] S301, determining a camera trajectory of the vehicle-mounted camera according to the postures corresponding to two adjacent key frames.

[0083] It can be understood that the posture corresponding to the key frame can be the posture of the key frame in the VO world system, and the posture is fixed.

[0084] It can be understood that the pose graph corresponding to the key frame output by the VO module is a sparse pose graph.

[0085] Exemplarily, by quickly densifying the VO pose based on the sparse pose graph, a dense VO pose graph can be obtained, and then based on the dense VO pose graph and the pose of the key frame, the pose of the ordinary frame can be optimized.

[0086] Figure 4 This is a densified VO pose graph provided in an embodiment of the present application. Figure 4 As shown, the densified VO pose graph includes all image frames captured by the above-mentioned on-board camera, and different numbers of ordinary frames can be contained between adjacent key frames.

[0087] In this step, a possible implementation method may be: obtaining a fourth residual according to the postures corresponding to two adjacent key frames; obtaining a posture corresponding to the ordinary frame between the two adjacent key frames according to the fourth residual; and determining the camera trajectory according to the order of the timestamps corresponding to the key frames and the ordinary frames.

[0088] Exemplarily, the fourth residual can be expressed by the following formula:

[0089] f 4 =(T_w_c j ) -1 T_w_c i T_c i _c j

[0090] Among them, f 4 represents the fourth residual, c i and c j represents adjacent key frames, T_w_c j Represents key frame c j The position of T_w_c i Represents key frame c i The position of T_c i _c j Represents two adjacent key frames c i and c j The relative displacement between .

[0091] For example, T_c i _c j It can be expressed by the following formula:

[0092] T_c i _c j =T_w_c i T_w_c j

[0093] For example, the pose of the key frame may be the transformation matrix of the vehicle-mounted camera in the VO world system.

[0094] For example, the pose of a normal frame can be expressed by the following formula:

[0095] T_w_c * =argmin∑f 4 2 ∑ -1

[0096] Among them, T_w_c * represents the pose of a normal frame, ∑ -1 represents the information matrix.

[0097] Exemplarily, the camera trajectory is the poses corresponding to all image frames arranged in time order.

[0098] S302, for each image frame in the camera trajectory, interpolate the vehicle trajectory according to the timestamp to obtain the position and posture of the target vehicle corresponding to the timestamp.

[0099] It can be understood that the vehicle trajectory and the camera trajectory are the corresponding trajectories of the target vehicle when it is traveling in the same period of time. Through the timestamp corresponding to each image frame in the camera trajectory, the position and posture of the target vehicle corresponding to the timestamp can be obtained in the vehicle trajectory.

[0100] It can be understood that the position and posture of the target vehicle is the position and posture of the target vehicle in the DR world system.

[0101] Exemplarily, the position and posture of the target vehicle is the position and posture of the vehicle system.

[0102] It should be noted that the vehicle system is determined in the following way: the rear axle center of the target vehicle is recorded as the vehicle system, the front direction is the x-axis, and the upward direction is the z-axis. Therefore, in general, the vehicle system only has a linear velocity and a steering angular velocity in the x-axis direction.

[0103] In an embodiment of the present application, the camera trajectory of the vehicle-mounted camera is determined according to the postures corresponding to two adjacent key frames, and the vehicle trajectory is interpolated according to the timestamp corresponding to each image frame in the camera trajectory to obtain the posture of the target vehicle, so as to realize the relative displacement of the vehicle-mounted camera and the target vehicle by constraining the posture of the target vehicle.

[0104] Optionally, the keyframe information may also include a ground plane equation.

[0105] Combine the following Figure 5 The specific implementation method of calibrating the rotational extrinsic parameters of the vehicle-mounted camera based on the key frame information, vehicle trajectory and the position and posture of the target vehicle is described in detail.

[0106] Figure 5 Schematic diagram of the process of calibrating the external parameters of the vehicle-mounted camera provided in the embodiment of the present application Figure 3 .like Figure 5 As shown, according to the key frame information, the vehicle trajectory and the position and posture of the target vehicle, the specific implementation method of calibrating the rotation external parameters of the vehicle-mounted camera may include the following steps:

[0107] S501, determining a first residual according to the postures corresponding to two adjacent key frames, the posture of the target vehicle, and the transformation matrix of the rotation extrinsic parameter to be calibrated.

[0108] The first residual is used to represent the relative displacement between the vehicle-mounted camera and the target vehicle.

[0109] Exemplarily, the first residual can be expressed by the following formula:

[0110] f 1 =T_b i _b j T_b_cT_c j _wT_w_c i (T_b_c) -1

[0111] Among them, f 1 represents the first residual, T_b_c represents the transformation matrix, T_c j _w indicates key frame c j The position of T_w_c i Represents key frame c i The position of T_b i _b j Represents key frame c i The corresponding timestamp corresponds to the target vehicle and key frame c j The corresponding timestamps correspond to the relative displacement between the target vehicles.

[0112] For example, T_b i _b j It can be expressed by the following formula:

[0113] T_b i _b j =T_b i _wT_w_b j

[0114] Among them, T_b i _w indicates key frame c i The inverse matrix of the target vehicle's pose at the corresponding timestamp, T_w_b j Represents key frame c j The pose of the target vehicle at the corresponding timestamp.

[0115] Among them, T_b i _w can be through T_w_b i Converted.

[0116] S502, determining a second residual according to the vehicle trajectory, the camera trajectory, the position and posture of the key frame, and the rotation matrix of the rotation extrinsic parameter to be calibrated.

[0117] The second residual is used to represent the relationship between the vehicle-mounted camera and the forward direction of the target vehicle, and the transformation matrix includes a rotation matrix.

[0118] Exemplarily, the transformation matrix may consist of two parts: a rotation matrix and a translation vector, and the transformation matrix may be a 4×4 matrix.

[0119] Exemplarily, the second residual can be expressed by the following formula:

[0120] f 2 =R_w_c i R_c_b[1;0;0]-V b

[0121] Among them, f 2 represents the second residual, R_w_c i Represents key frame c i The rotation matrix in the VO world system, R_c_b represents the rotation matrix of the rotation external parameter to be calibrated, [1; 0; 0] represents the unit vector in the x-axis direction of the vehicle system, V b Represents the unit vector obtained by normalizing the velocity vector.

[0122] For example, R_w_c i You can use keyframe c i The pose transformation is obtained.

[0123] In this step, a possible implementation method may be: screening the vehicle trajectory to determine the trajectory of the uniform straight line in the vehicle trajectory as the target vehicle trajectory; based on the start timestamp and end timestamp corresponding to the target vehicle trajectory, according to the camera trajectory, determining the 3D point position corresponding to the vehicle-mounted camera; performing spline fitting on the 3D point position to obtain the velocity vector of the vehicle-mounted camera in the world system; and determining the second residual according to the velocity vector and the rotation matrix.

[0124] Exemplarily, for the vehicle trajectory, all trajectory segments with bumps, strong pitch changes, obvious steering, and obvious acceleration in the vehicle trajectory are removed, and only the trajectory segments of uniform straight lines are retained, and the trajectory segments of uniform straight lines are used as the target vehicle trajectory.

[0125] Exemplarily, based on the start timestamp and the end timestamp corresponding to the target vehicle body trajectory, the target camera trajectory corresponding to the start timestamp and the end timestamp respectively is determined in the camera trajectory, and a cubic spline fitting is performed on the 3D point position (x, y, z) corresponding to the vehicle-mounted camera contained in the target camera trajectory to obtain the velocity vector of the vehicle-mounted camera in the world system.

[0126] It is understandable that, because the camera trajectory is a densified trajectory, the uniform straight line trajectory segments selected from the vehicle trajectory can be fitted with spline to ensure high accuracy. Under the assumption of zero angular velocity and rigid body, no matter how the vehicle system from the vehicle camera to the target vehicle rotates, the linear velocity direction of the center of the vehicle camera is always equal to the linear velocity direction of the vehicle system.

[0127] S503: Determine a third residual according to the ground plane equation, the position and posture of the key frame, and the rotation matrix.

[0128] Among them, the third residual is used to represent the relationship between the vehicle-mounted camera and the ground.

[0129] Exemplarily, the third residual can be expressed by the following formula:

[0130] f 3 =R_w_c i R_c_b[0;0;1]-N i

[0131] Among them, f 3 represents the third residual, [0; 0; 1] represents the unit vector in the z-axis direction of the vehicle system, N i Represents key frame c i The unit vector obtained by normalizing the upward normal vector of the corresponding ground plane equation.

[0132] S504: Calibrate the rotation extrinsic parameters of the vehicle-mounted camera according to the first residual, the second residual, and the third residual.

[0133] Exemplarily, the rotation extrinsic parameter can be expressed by the following formula:

[0134] R_c_b=argmin(∑f 1 2 ∑ -1 +∑f 2 2 ∑ -1 +∑f 3 2 ∑ -1 )

[0135] In the embodiment of the present application, a first residual for representing the relative displacement between the vehicle-mounted camera and the target vehicle is determined according to the postures corresponding to two adjacent key frames, the posture of the target vehicle, and the transformation matrix of the rotation extrinsic parameter to be calibrated, so as to realize the constraint on the relative displacement between the vehicle-mounted camera and the target vehicle. A second residual for representing the relationship between the vehicle-mounted camera and the forward direction of the target vehicle is determined according to the vehicle trajectory, the camera trajectory, the posture of the key frames, and the rotation matrix of the rotation extrinsic parameter to be calibrated. By combining the constraint on the relationship between the vehicle-mounted camera and the forward direction of the target vehicle and the constraint on the relative displacement, the relative displacement provides a better constraint in the case of more pitching or turning. When driving at a constant speed in a straight line, the x-axis direction of the target vehicle can be accurately estimated, forming a direct constraint on the rotation extrinsic parameters. The third residual used to represent the relationship between the vehicle camera and the ground is determined according to the ground plane equation, the position and rotation matrix of the key frame, so as to realize the constraint on the degree of freedom. The ground plane equation of the key frame can be used to realize the constraint on the Z-axis direction under the vehicle body. By constraining the relative displacement between the vehicle camera and the target vehicle, the relationship between the vehicle camera and the forward direction of the target vehicle, and the relationship between the vehicle camera and the ground, the calibration of the vehicle camera extrinsic parameters can be realized in scenarios where the target vehicle's driving trajectory is not limited, and the calibration accuracy can be improved at the same time.

[0136] Optionally, the ground plane equation corresponding to the key frame can be obtained in the following way: obtain the key frame in the sliding window, in which the road surface features are marked; for each key frame in the sliding window, determine the preset area according to the position and posture of the key frame; perform plane fitting processing on the 3D features corresponding to the road surface features in the preset area to obtain the ground plane equation.

[0137] In one possible implementation, for each key frame in the sliding window, a circular preset area is made with the posture corresponding to the key frame as the center and a preset length as the radius, the road features contained in the preset area are determined, and the ground plane equation corresponding to the key frame is fitted based on the 3D features corresponding to the road features.

[0138] Exemplarily, the 3D feature may be a 3D position corresponding to a 3D feature point.

[0139] It can be understood that the above embodiment is for calibrating the extrinsic parameters of a vehicle-mounted camera in the target vehicle. However, usually, the number of vehicle-mounted cameras of the target vehicle, that is, the number of cameras of the automatic driving system in the target vehicle, is greater than one and there is a common view area. In this case, visual constraints between vehicle-mounted cameras can also be established to further improve the extrinsic parameter accuracy of the vehicle-mounted camera.

[0140] In a possible implementation, multiple VO modules are first merged into a unified world system, and then the rotation extrinsic parameters of each vehicle-mounted camera are optimized again under the unified world system.

[0141] For example, one implementation method of merging multiple VO modules into a unified world system is as follows: at the VO pose graph level, the transformation matrix (T_b_c) of the rotation extrinsic parameter obtained in the previous step and the relative displacement (T_c i _c j ), optimize the coordinates of all on-board cameras in the unified world system; use the conversion relationship from the original 3D point and line features to the key frame to convert the 3D points and lines of all VO modules to the unified world system; project the 3D point and line features to the cross-camera coordinate system, match the features within a certain reprojection range, and if successful, establish a new cross-camera connection; perform global BA optimization, which is different from the back end in the above VO module. Due to the existence of multiple on-board cameras and the determination of the external parameter translation, the scale is constrained by using the arm length between cameras, combined with the reprojection error for optimization, to achieve the reconstruction of the camera pose and 3D features in the same world system.

[0142] For example, the specific implementation method of re-optimizing the rotation extrinsic parameters of each vehicle-mounted camera under the unified world system is similar to the above. In the step of estimating the ground plane equation of the key frame, it is no longer necessary to iteratively update the initial position of the 3D feature and the ground plane equation, but directly estimate the ground plane shared by multiple vehicle-mounted cameras in the unified world system reconstructed in the previous step.

[0143] It is understandable that the constraints of the camera can make the reconstruction result more accurate than the single-purpose VO. This step can improve the calibration accuracy of the rotation extrinsic parameters.

[0144] In summary, the method for calibrating the extrinsic parameters of a vehicle-mounted camera provided in the embodiment of the present application has the following beneficial effects:

[0145] 1) It does not rely on specific road signs, such as some lane line recognition schemes, but adopts universal point and line feature extraction, and any road signs, textures, traces, etc. can be used;

[0146] 2) Less reliance on road smoothness, more accurate ground plane estimation. By making a plane assumption only in the vicinity of the vehicle body, multi-view observations are integrated in the VO module and optimization to make ground plane estimation more accurate. At the same time, the use of the front and rear driving trajectories and other observations in the reconstruction can better filter out ground planes that do not meet the requirements, which is more accurate than methods based on single-frame ground planes.

[0147] 3) It does not rely on a specific driving trajectory, can produce good constraints for both straight-line driving and turning, and is universal;

[0148] 4) There is no limit on the number of vehicle-mounted cameras, and it is applicable to any number of vehicle-mounted cameras;

[0149] 5) Use ground reconstruction and vehicle motion as dual constraints to improve calibration results;

[0150] 6) Utilize the constraints between multiple on-board cameras to perform global reconstruction and extrinsic parameter optimization to improve the accuracy of rotation extrinsic parameters.

[0151] The following is an embodiment of the device of the present application, which can be used to execute the embodiment of the method of the present application. For details not disclosed in the embodiment of the device of the present application, please refer to the embodiment of the method of the present application.

[0152] Figure 6 This is a schematic diagram of the structure of a calibration device for an on-board camera external parameter provided in an embodiment of the present application. Figure 6 As shown, the vehicle-mounted camera external parameter calibration device 60 includes an acquisition module 610, a processing module 620 and a calibration module 630.

[0153] The acquisition module 610 is used to acquire the vehicle body trajectory of the target vehicle and the image frames captured by the vehicle-mounted camera in the same period;

[0154] The processing module 620 is used to input the image frame into the VO module and obtain the key frame information output by the VO module;

[0155] The calibration module 630 is used to calibrate the rotation extrinsic parameters of the vehicle-mounted camera according to the key frame information and the vehicle trajectory.

[0156] In a possible implementation, the calibration module 630 is specifically used to: determine the position and posture of the target vehicle according to the key frame information and the vehicle trajectory; and calibrate the rotation extrinsic parameters of the vehicle-mounted camera according to the key frame information, the vehicle trajectory and the position and posture of the target vehicle.

[0157] In one possible implementation, the key frame information includes different key frames and the posture corresponding to each key frame. The calibration module 630 is also used to: determine the camera trajectory of the vehicle-mounted camera according to the postures corresponding to two adjacent key frames; for the timestamp corresponding to each image frame in the camera trajectory, interpolate the vehicle trajectory according to the timestamp to obtain the posture of the target vehicle corresponding to the timestamp.

[0158] In a possible implementation, the key frame information also includes a ground plane equation corresponding to the key frame, and the calibration module 630 is also used to: determine a first residual based on the postures corresponding to two adjacent key frames, the posture of the target vehicle, and the transformation matrix of the rotation extrinsic parameter to be calibrated, the first residual is used to represent the relative displacement of the vehicle-mounted camera and the target vehicle; determine a second residual based on the vehicle trajectory, the camera trajectory, the posture of the key frame, and the rotation matrix of the rotation extrinsic parameter to be calibrated, the second residual is used to represent the relationship between the vehicle-mounted camera and the forward direction of the target vehicle, and the transformation matrix includes a rotation matrix; determine a third residual based on the ground plane equation, the posture of the key frame, and the rotation matrix, the third residual is used to represent the relationship between the vehicle-mounted camera and the ground; calibrate the rotation extrinsic parameters of the vehicle-mounted camera based on the first residual, the second residual, and the third residual.

[0159] In a possible implementation, the calibration module 640 is also used to: screen the vehicle trajectory to determine that the trajectory of the uniform straight line in the vehicle trajectory is the target vehicle trajectory; based on the start timestamp and end timestamp corresponding to the target vehicle trajectory, determine the 3D point position corresponding to the vehicle-mounted camera according to the camera trajectory; perform spline fitting on the 3D point position to obtain the velocity vector of the vehicle-mounted camera in the world system; and determine the second residual according to the velocity vector and the rotation matrix.

[0160] In one possible implementation, the ground plane equation is obtained in the following manner: obtaining a key frame and a 3D feature in a sliding window, wherein the key frame is marked with road features; for each key frame in the sliding window, determining a preset area according to the position and posture of the key frame; performing plane fitting processing on the 3D features corresponding to the road features in the preset area to obtain the ground plane equation.

[0161] In a possible implementation, the calibration module 630 is further used to: obtain a fourth residual based on the postures corresponding to two adjacent key frames; obtain a posture corresponding to the ordinary frame between the two adjacent key frames based on the fourth residual; and determine the camera trajectory according to the order of the timestamps corresponding to the key frames and the ordinary frames.

[0162] The device for calibrating the extrinsic parameters of a vehicle-mounted camera provided in this embodiment can be used to execute the method steps of the above-mentioned method for calibrating the extrinsic parameters of a vehicle-mounted camera. The specific implementation method and technical effect are similar and will not be repeated here.

[0163] Figure 7 This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. Figure 7 As shown, the electronic device 70 includes at least one processor 710 and a memory 720. Optionally, the electronic device 70 also includes a communication interface 730. The processor 710, the memory 720 and the communication interface 730 are connected via a system bus 740.

[0164] In a specific implementation process, at least one processor 710 executes the computer execution instructions stored in the memory 720, so that at least one processor 710 executes the above method.

[0165] The specific implementation process of the processor 710 can be found in the above method embodiment, and its implementation principle and technical effect are similar, so this embodiment will not be repeated here.

[0166] In the above embodiments, it should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in the invention may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0167] The memory may include a high-speed memory (Random Access Memory, referred to as RAM), and may also include a non-volatile memory (Non-volatile Memory, referred to as NVM), such as at least one disk memory.

[0168] The system bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The system bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the system bus in the drawings of the present application is not limited to only one bus or one type of bus.

[0169] The present application also provides a computer program product, including a computer program, which implements the above method when executed by a processor.

[0170] The present application also provides a computer-readable storage medium, in which computer-executable instructions are stored. When a processor executes the computer-executable instructions, the above method is implemented.

[0171] The above-mentioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk. The readable storage medium can be any available medium that can be accessed by a general or special-purpose computer.

[0172] An exemplary readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can be located in an application specific integrated circuit (Application Specific Integrated Circuits, referred to as ASIC). Of course, the processor and the readable storage medium can also exist in the device as discrete components.

[0173] The division of units is only a logical function division, and there may be other divisions in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interface, device or unit, which can be electrical, mechanical or other forms.

[0174] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0175] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0176] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods of each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store program codes.

[0177] Those skilled in the art can understand that all or part of the steps of implementing the above-mentioned method embodiments can be completed by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above-mentioned method embodiments are executed; and the aforementioned storage medium includes: ROM, RAM, disk or optical disk and other media that can store program codes.

[0178] Finally, it should be noted that those skilled in the art will readily conceive of other embodiments of the present invention after considering the specification and practicing the invention disclosed herein. The present invention is intended to cover any variations, uses or adaptations of the present invention, which follow the general principles of the present invention and include common knowledge or customary technical means in the art not disclosed by the present invention, are not limited to the precise structure described above and shown in the drawings, and may be modified and changed in various ways without departing from the scope thereof. The scope of the present invention is limited only by the appended claims.

Claims

1. A method for calibrating the external parameters of a vehicle-mounted camera, characterized in that: include: Obtain the body trajectory of the target vehicle and the image frames captured by the vehicle-mounted camera in the same period; Input the image frame into a visual odometer VO module to obtain key frame information output by the VO module; The rotation extrinsic parameters of the vehicle-mounted camera are calibrated according to the key frame information and the vehicle trajectory.

2. The calibration method according to claim 1, characterized in that: The step of calibrating the rotational external parameters of the vehicle-mounted camera according to the key frame information and the vehicle trajectory includes: Determine the position and posture of the target vehicle according to the key frame information and the vehicle trajectory; The rotation extrinsic parameters of the vehicle-mounted camera are calibrated according to the key frame information, the vehicle trajectory and the position and posture of the target vehicle.

3. The calibration method according to claim 2, characterized in that: The key frame information includes different key frames and a posture corresponding to each key frame, and determining the posture of the target vehicle according to the key frame information and the vehicle body trajectory includes: Determining the camera trajectory of the vehicle-mounted camera according to the poses corresponding to two adjacent key frames; For each time stamp corresponding to each image frame in the camera trajectory, the vehicle trajectory is interpolated according to the time stamp to obtain the position and posture of the target vehicle corresponding to the time stamp.

4. The calibration method according to claim 3, characterized in that: The key frame information also includes a ground plane equation corresponding to the key frame. The calibrating the rotation extrinsic parameters of the vehicle-mounted camera according to the key frame information, the vehicle trajectory, and the position and posture of the target vehicle includes: Determine a first residual according to the postures corresponding to two adjacent key frames, the posture of the target vehicle, and the transformation matrix of the rotation extrinsic parameter to be calibrated, wherein the first residual is used to represent the relative displacement between the vehicle-mounted camera and the target vehicle; Determine a second residual according to the vehicle trajectory, the camera trajectory, the position and posture of the key frame, and the rotation matrix of the rotation extrinsic parameter to be calibrated, wherein the second residual is used to represent the relationship between the vehicle-mounted camera and the forward direction of the target vehicle, and the transformation matrix includes the rotation matrix; Determine a third residual according to the ground plane equation, the position and posture of the key frame and the rotation matrix, wherein the third residual is used to represent the relationship between the vehicle-mounted camera and the ground; The rotation extrinsic parameter of the vehicle-mounted camera is calibrated according to the first residual, the second residual, and the third residual.

5. The calibration method according to claim 4, characterized in that: The determining of the second residual according to the vehicle trajectory, the camera trajectory, the position and posture of the key frame and the rotation matrix of the rotation extrinsic parameter to be calibrated includes: Screening the vehicle trajectory to determine a uniform straight line trajectory in the vehicle trajectory as a target vehicle trajectory; Based on the start timestamp and the end timestamp corresponding to the target vehicle trajectory, and according to the camera trajectory, determine the 3D point position corresponding to the vehicle-mounted camera; Performing spline fitting processing on the 3D point position to obtain a velocity vector of the vehicle-mounted camera in the world system; The second residual is determined according to the velocity vector and the rotation matrix.

6. The calibration method according to claim 4, characterized in that: The ground plane equation is obtained as follows: Acquire a key frame in the sliding window, wherein the key frame is marked with road features; For each key frame in the sliding window, determining a preset area according to the position and posture of the key frame; A plane fitting process is performed on the 3D features corresponding to the road surface features in the preset area to obtain the ground plane equation.

7. The calibration method according to any one of claims 3 to 6, characterized in that: The step of determining the camera trajectory of the vehicle-mounted camera according to the poses respectively corresponding to two adjacent key frames includes: According to the postures corresponding to two adjacent key frames, the fourth residual is obtained; According to the fourth residual, a posture corresponding to a common frame between two adjacent key frames is obtained; The camera trajectory is determined according to the sequence of timestamps corresponding to the key frames and the ordinary frames.

8. A device for calibrating external parameters of a vehicle-mounted camera, characterized in that: include: An acquisition module is used to acquire the body trajectory of the target vehicle and the image frames captured by the vehicle-mounted camera within the same period of time; A processing module, used for inputting the image frame into a visual odometer VO module to obtain key frame information output by the VO module; A calibration module is used to calibrate the rotation extrinsic parameters of the vehicle-mounted camera according to the key frame information and the vehicle trajectory.

9. An electronic device, characterized in that: include: at least one processor; and a memory communicatively coupled to the at least one processor; The memory is used to store instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, which are used to implement the method according to any one of claims 1 to 7 when executed by a processor.

11. A computer program product, characterized in that include: A computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images