External parameter correction method, device, electronic device and storage medium for vehicle-mounted camera

Abstract

The present application discloses a method, device, electronic device, and storage medium for extrinsic parameter correction of a vehicle-mounted camera, wherein the method includes: obtaining a road image captured by the vehicle-mounted camera, and using a preset road surface recognition model to recognize the road image to obtain a road surface recognition result; constructing an extrinsic parameter correction constraint condition based on the road surface recognition result, and constructing a nonlinear optimization model based on the extrinsic parameter correction constraint condition and a predefined coordinate system; using the nonlinear optimization model, determining the correction extrinsic parameter of the vehicle-mounted camera; obtaining the extrinsic parameter to be corrected of the vehicle-mounted camera, and using the corrected extrinsic parameter to correct the extrinsic parameter to be corrected of the vehicle-mounted camera. The present application does not require the artificial setting of markers on the roadside for extrinsic parameter correction, and uses the extrinsic parameter correction constraint condition determined based on the recognition result of the road image to construct a nonlinear optimization model and solve the optimal extrinsic parameter, thereby realizing real-time correction of the external parameter during vehicle driving, and also has good robustness for scenes with general road flatness.

Description

Technical Field

[0001] The present application relates to the technical field of camera calibration, and in particular to an external parameter calibration method, device, electronic device, and storage medium for a vehicle-mounted camera. Background Art

[0002] In the image measurement process and machine vision applications, in order to determine the relationship between the three-dimensional geometric position of a point on the surface of a spatial object and its corresponding point in the image, a geometric model of camera imaging must be established. These geometric model parameters are the camera parameters. Under most conditions, these parameters must be obtained through experiments and calculations. The process of solving the parameters is called camera calibration. Whether in image measurement or machine vision applications, the calibration of camera parameters is a very critical link. The accuracy of the calibration results and the stability of the algorithm directly affect the accuracy of the results produced by the camera.

[0003] In the scenario of autonomous driving, the calibration of the parameters of the on-board camera is also particularly important. The quality of the calibration results directly affects the positioning accuracy and positioning stability of the subsequent autonomous driving vehicle. At present, the external parameters of most on-board cameras, that is, the external parameters of the on-board camera in the vehicle coordinate system, are mainly calibrated offline or online using a calibration board in a certain area. However, the vehicle will face up and down bumps during driving, which will also affect the external parameters of the on-board camera.

[0004] When positioning a vehicle based on a high-precision map, it is necessary to use the external parameters of the on-board camera to transfer the road features in the image to the vehicle coordinate system, and then match them with the road features in the high-precision map for positioning. Therefore, changes in external parameters will cause large errors in the position and scale of road features, which will in turn cause larger positioning errors.

[0005] In order to avoid the above problems, a solution is needed to achieve real-time correction of the external parameters of the vehicle-mounted camera. However, the existing methods cannot meet this requirement and have poor flexibility. Summary of the invention

[0006] The embodiments of the present application provide a method, device, electronic device, and storage medium for calibrating the external parameters of a vehicle-mounted camera to calibrate the external parameters of the vehicle-mounted camera in real time.

[0007] The present application embodiment adopts the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides an external parameter calibration method for a vehicle-mounted camera, wherein the method comprises:

[0009] Acquire a road image captured by a vehicle-mounted camera, and use a preset road surface recognition model to recognize the road image to obtain a road surface recognition result;

[0010] Constructing an external parameter correction constraint condition according to the road surface recognition result, and constructing a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system;

[0011] Determining the correction extrinsic parameters of the vehicle-mounted camera using the nonlinear optimization model;

[0012] The extrinsic parameters to be corrected of the vehicle-mounted camera are obtained, and the corrected extrinsic parameters are used to correct the extrinsic parameters to be corrected of the vehicle-mounted camera.

[0013] Optionally, the predefined coordinate system includes a vehicle body coordinate system, the extrinsic parameter correction constraint condition includes a ground height residual constraint condition, and constructing the extrinsic parameter correction constraint condition according to the road surface recognition result includes:

[0014] Determining an estimate of the ground height residual constraint condition according to the road surface recognition result;

[0015] Determine the observed value of the ground height residual constraint condition according to the preset ground height value in the vehicle body coordinate system;

[0016] The ground height residual constraint condition is determined according to a difference between an estimated amount of the ground height residual constraint condition and an observed amount of the ground height residual constraint condition.

[0017] Optionally, the predefined coordinate system further includes a camera coordinate system, the road surface recognition result includes coordinates of road surface pixel points in the camera coordinate system, and determining the estimated amount of the ground height residual constraint condition according to the road surface recognition result includes:

[0018] The coordinates of the road pixel points in the camera coordinate system are converted to the vehicle coordinate system to obtain the coordinates of the road pixel points in the vehicle coordinate system, wherein the coordinates of the road pixel points in the vehicle coordinate system include a Z-axis coordinate value in a direction perpendicular to the ground;

[0019] The Z-axis coordinate value is used as an estimate of the ground height residual constraint condition.

[0020] Optionally, the extrinsic parameter correction constraint condition further includes an initial extrinsic parameter constraint condition, and the initial extrinsic parameter constraint condition includes multiple initial extrinsic parameters and weights corresponding to each initial extrinsic parameter, and the initial extrinsic parameter constraint condition is determined in the following manner:

[0021] Acquire multiple initial external parameters of the vehicle-mounted camera;

[0022] According to the type of each initial external parameter, the weight of each initial external parameter is determined.

[0023] Optionally, the external parameters include translation, yaw angle, roll angle and pitch angle, and determining the weight of each initial external parameter according to the type of each initial external parameter includes:

[0024] Setting a first weight for the translation amount and the yaw angle;

[0025] Setting a second weight for the roll angle and the pitch angle;

[0026] The first weight is greater than the second weight.

[0027] Optionally, the determining the correction extrinsic parameters of the vehicle-mounted camera by using the nonlinear optimization model includes:

[0028] The nonlinear optimization model is solved using a preset gradient descent algorithm to obtain correction extrinsic parameters of the vehicle-mounted camera.

[0029] Optionally, the extrinsic parameters to be corrected include an initial roll angle and an initial pitch angle, the corrected extrinsic parameters include a corrected roll angle and a corrected pitch angle, and the correcting the extrinsic parameters to be corrected of the vehicle-mounted camera using the corrected extrinsic parameters includes:

[0030] The initial roll angle is replaced by the corrected roll angle, and the initial pitch angle is replaced by the corrected pitch angle.

[0031] In a second aspect, an embodiment of the present application further provides an external parameter calibration device for a vehicle-mounted camera, wherein the device comprises:

[0032] An acquisition unit is used to acquire a road image captured by a vehicle-mounted camera, and recognize the road image using a preset road recognition model to obtain a road recognition result;

[0033] A construction unit, configured to construct an external parameter correction constraint condition according to the road surface recognition result, and to construct a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system;

[0034] A determination unit, configured to determine a correction extrinsic parameter of the vehicle-mounted camera using the nonlinear optimization model;

[0035] The correction unit is used to obtain the extrinsic parameters to be corrected of the vehicle-mounted camera, and to correct the extrinsic parameters to be corrected of the vehicle-mounted camera using the correction extrinsic parameters.

[0036] In a third aspect, an embodiment of the present application further provides an electronic device, including:

[0037] Processor; and

[0038] A memory arranged to store computer executable instructions which, when executed, cause the processor to perform any of the methods described above.

[0039] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes any of the aforementioned methods.

[0040] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: the extrinsic parameter correction method of the vehicle-mounted camera in the embodiments of the present application first obtains the road image captured by the vehicle-mounted camera, and uses the preset road surface recognition model to recognize the road image to obtain the road surface recognition result; then constructs the extrinsic parameter correction constraint conditions according to the road surface recognition result, and constructs a nonlinear optimization model according to the extrinsic parameter correction constraint conditions and the predefined coordinate system; then uses the nonlinear optimization model to determine the correction extrinsic parameters of the vehicle-mounted camera; finally, obtains the extrinsic parameters to be corrected of the vehicle-mounted camera, and uses the corrected extrinsic parameters to correct the extrinsic parameters to be corrected of the vehicle-mounted camera. The extrinsic parameter correction method of the vehicle-mounted camera in the embodiments of the present application does not require the artificial setting of markers on the roadside for extrinsic parameter correction. It uses the extrinsic parameter correction constraint conditions determined based on the recognition result of the road image to construct a nonlinear optimization model and solve the optimal extrinsic parameters, thereby realizing the real-time correction of the extrinsic parameters during vehicle driving, and also has good robustness for scenes with general road flatness. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0042] Figure 1 A schematic diagram of a flow chart of an external parameter calibration method for a vehicle-mounted camera in an embodiment of the present application;

[0043] Figure 2 This is a schematic diagram of the structure of an external parameter calibration device for a vehicle-mounted camera in an embodiment of the present application;

[0044] Figure 3 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0046] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0047] The present application embodiment provides an external parameter calibration method for a vehicle-mounted camera, such as Figure 1 As shown, a flow chart of an external parameter calibration method for a vehicle-mounted camera in an embodiment of the present application is provided, and the method at least includes the following steps S110 to S140:

[0048] Step S110, obtaining a road image captured by a vehicle-mounted camera, and using a preset road surface recognition model to recognize the road image to obtain a road surface recognition result.

[0049] When calibrating the external parameters of the vehicle-mounted camera, the embodiment of the present application can first obtain a road image taken by the vehicle-mounted camera. The road image may refer to an image obtained by the vehicle-mounted camera taking pictures of the road surface and vehicles on the road surface within the surrounding visible range during the driving process of the vehicle.

[0050] After obtaining the road image, the road image is recognized using a pre-trained road surface recognition model, which can be specifically the road surface itself or the road signs on the road surface such as lane lines, stop lines or arrows, so as to obtain the road surface recognition result. The above-mentioned road surface recognition model can be obtained based on the existing convolutional neural network training, such as the LaneNet model. Of course, the specific form of the road surface recognition model can be flexibly selected by those skilled in the art according to actual needs, and is not specifically limited here.

[0051] Step S120, constructing external parameter correction constraints according to the road surface recognition result, and constructing a nonlinear optimization model according to the external parameter correction constraints and a predefined coordinate system.

[0052] After obtaining the road surface recognition result, it is necessary to construct external parameter correction constraints based on the recognized road surface features. The external parameter correction constraints here can be used to constrain the optimization process of the external parameters. Then, a nonlinear optimization model is constructed based on the constructed external parameter correction constraints combined with the predefined coordinate system. The predefined coordinate system can specifically include a vehicle body coordinate system and a camera coordinate system. The specific form can be flexibly defined according to the actual application scenario or actual needs. The external parameters of the embodiment of the present application can be understood as the conversion relationship from the camera coordinate system to the vehicle body coordinate system.

[0053] The nonlinear optimization model can be used to solve the optimal value of the external parameter, which can be implemented by using the least squares algorithm. The principle of the least squares algorithm is to find the best function matching of the data by minimizing the sum of squares of the errors. Of course, those skilled in the art can also flexibly select other types of nonlinear optimization models according to actual needs, which are not specifically limited here.

[0054] Step S130, using the nonlinear optimization model to determine the correction extrinsic parameters of the vehicle-mounted camera.

[0055] After the nonlinear optimization model is constructed, the preset solution algorithm can be used to solve the nonlinear optimization model, that is, the output results of the model are gradually converged through multiple iterations to obtain the current optimal external parameters as the correction external parameters of the vehicle-mounted camera.

[0056] Step S140, obtaining the extrinsic parameters to be calibrated of the vehicle-mounted camera, and calibrating the extrinsic parameters to be calibrated of the vehicle-mounted camera using the calibration extrinsic parameters.

[0057] Since the vehicle will first perform an initial calibration of the camera extrinsic parameters during the initialization phase, the initially calibrated extrinsic parameters may change during the driving process of the vehicle, and the camera extrinsic parameters need to be corrected. Therefore, after obtaining the corrected extrinsic parameters, the embodiment of the present application can first obtain the extrinsic parameters to be corrected that have been calibrated by the on-board camera during the initialization phase, and then use the corrected extrinsic parameters to correct them, so as to obtain more accurate camera extrinsic parameters, thereby providing more accurate data support for other autonomous driving modules.

[0058] The extrinsic parameter correction method for the vehicle-mounted camera in the embodiment of the present application does not require the artificial setting of roadside markers for extrinsic parameter correction. It uses the extrinsic parameter correction constraints determined based on the recognition results of the road image to construct a nonlinear optimization model and solve the optimal extrinsic parameters, thereby realizing real-time correction of external parameters during vehicle driving, and also has good robustness for scenes with general road flatness.

[0059] In one embodiment of the present application, the predefined coordinate system includes a vehicle body coordinate system, the extrinsic parameter correction constraint includes a ground height residual constraint, and the construction of the extrinsic parameter correction constraint based on the road surface recognition result includes: determining an estimate of the ground height residual constraint based on the road surface recognition result; determining an observed value of the ground height residual constraint based on a preset ground height value in the vehicle body coordinate system; and determining the ground height residual constraint based on the difference between the estimate of the ground height residual constraint and the observed value of the ground height residual constraint.

[0060] As mentioned above, the predefined coordinate system of the embodiment of the present application includes a vehicle body coordinate system. Specifically, the xyz axis directions can be defined as front-left-top, respectively, and the projection of the rear axle center of the vehicle on the ground is used as the origin of the vehicle body coordinate system.

[0061] The external parameter correction constraint of the embodiment of the present application may include a ground height residual constraint, which is mainly set based on a relatively fixed starting point of the ground height value in the vehicle body coordinate system. For example, when the vehicle is moving close to the ground, the ground height value corresponding to the Z axis can be assumed to be 0 or a value close to 0 (centimeter level) in the vehicle body coordinate system, and this value can be used as the observed value Z. o At the same time, the estimated value of the ground height residual constraint condition is determined according to the road surface recognition result output by the aforementioned road surface recognition model, that is, the estimated value of the ground height recognized by the road surface recognition model. Finally, the ground height residual constraint condition is constructed according to the difference between the estimated value and the observed value. One of the purposes of using the ground height residual constraint condition to constrain the model iteration is to make the difference between the above-mentioned estimated value and the observed value as close to 0 as possible, so as to obtain more accurate camera extrinsic parameters.

[0062] In one embodiment of the present application, the predefined coordinate system also includes a camera coordinate system, the road surface recognition result includes the coordinates of road surface pixel points in the camera coordinate system, and the determination of the estimate of the ground height residual constraint condition based on the road surface recognition result includes: converting the coordinates of the road surface pixel points in the camera coordinate system to the vehicle body coordinate system to obtain the coordinates of the road surface pixel points in the vehicle body coordinate system, the coordinates of the road surface pixel points in the vehicle body coordinate system include Z-axis coordinate values ​​in a direction perpendicular to the ground; and using the Z-axis coordinate value as the estimate of the ground height residual constraint condition.

[0063] As mentioned above, the predefined coordinate system of the embodiment of the present application may also include a camera coordinate system. Specifically, the xyz axis directions may be defined as right-bottom-front, respectively, and the camera optical center may be used as the origin of the camera coordinate system.

[0064] The road surface recognition result of the embodiment of the present application may specifically include the coordinates (x cam ,y cam ,z cam ), due to the above-mentioned ground height residual constraint condition, the observed quantity Z o is determined in the vehicle body coordinate system, so the embodiment of the present application determines the estimated value Z e When , we can first convert the coordinates of the road pixel points in the camera coordinate system to the vehicle coordinate system, that is, the unified coordinate system, so as to obtain the coordinates of the road pixel points in the vehicle coordinate system (x car ,y car ,z car ), since when constructing the ground height residual constraint condition, we only need to pay attention to the height value in the direction perpendicular to the ground in the vehicle coordinate system, so we can directly use the Z-axis coordinate value of the road pixel point in the above vehicle coordinate system as the estimated value of the ground height residual constraint condition, that is, z car =Z e , and finally Z e -Z o As a ground height residual constraint.

[0065] When converting the coordinates of the road pixel points in the camera coordinate system to the vehicle coordinate system, the following methods can be used:

[0066] 1) Calculate the depth d of the road pixel in the camera coordinate system:

[0067] d=f*H / y, (1)

[0068] Among them, d is the depth of the road pixel in the camera coordinate system, f is the focal length of the camera (in pixels), H is the height of the camera from the ground (in meters), and y is the projection of the road pixel in the image (in pixels).

[0069] 2) Calculate the coordinates of road pixel points in the vehicle coordinate system

[0070]

[0071] Where d is the depth of the road pixel in the camera coordinate system, R is the rotation matrix from the camera coordinate system to the vehicle coordinate system, t is the translation vector from the camera coordinate system to the vehicle coordinate system, K is the camera intrinsic parameter matrix, and K -1 Represents the inverse of the camera intrinsic parameter matrix, [u,v,1] T Represents homogeneous pixel coordinates, [X car ,Y car ,Z car ] T Represents the coordinates of the ground point in the vehicle coordinate system.

[0072] In one embodiment of the present application, the extrinsic parameter correction constraint also includes an initial extrinsic parameter constraint, and the initial extrinsic parameter constraint includes multiple initial extrinsic parameters and the weights corresponding to each initial extrinsic parameter. The initial extrinsic parameter constraint is determined in the following manner: obtaining multiple initial extrinsic parameters of the vehicle-mounted camera; and determining the weight of each initial extrinsic parameter according to the type of each initial extrinsic parameter.

[0073] The external parameter correction constraint conditions of the embodiment of the present application also include initial external parameter constraints. The initial external parameter constraints can be regarded as using the initial calibrated camera external parameters as a priori conditions to constrain the iterative process of the model. Although the external parameters will change during the movement of the vehicle, causing the initially calibrated external parameters to be no longer accurate, this change will not be large. Therefore, introducing the initial external parameters for further constraints can greatly improve the speed of model convergence.

[0074] Specifically, camera extrinsics generally include multiple parameters, such as translation (x, y), yaw, roll, and pitch, etc. Different types of extrinsics can be assigned different weights, so that different degrees of optimization can be performed based on the weights during the iteration process. For example, since the vehicle is moving near the ground, that is, in the horizontal direction, the size of the ground height residual, that is, the deviation in the direction perpendicular to the ground, obviously cannot be used to constrain the optimization of the extrinsic parameters in the horizontal direction. Therefore, based on this, different weights can be used to distinguish the optimization of different types of camera extrinsics.

[0075] In one embodiment of the present application, the external parameters include translation, yaw, roll and pitch, and determining the weights of each initial external parameter according to the type of each initial external parameter includes: setting a first weight for the translation and the yaw; setting a second weight for the roll and the pitch; wherein the first weight is greater than the second weight.

[0076] As mentioned above, since the size of the ground height residual cannot be used to constrain the optimization of external parameters in the horizontal direction, or cannot be accurately optimized, the embodiment of the present application can set smaller information weights for the roll angle and pitch angle of the rotating part. The smaller the weight, the greater the error of the corresponding external parameter, and the more emphasis needs to be placed on optimization. For external parameters in the horizontal direction such as translation and yaw angle, a larger weight can be set so that they can be kept as unchanged as possible during the optimization process.

[0077] It should be noted that the first weight set for the translation amount and yaw angle can be the same value or two different values. Similarly, the second weight set for the roll angle and pitch angle can be the same value or two different values, as long as the first weight is sufficiently greater than the second weight.

[0078] In one embodiment of the present application, determining the correction extrinsic parameters of the vehicle-mounted camera using the nonlinear optimization model includes: solving the nonlinear optimization model using a preset gradient descent algorithm to obtain the correction extrinsic parameters of the vehicle-mounted camera.

[0079] When solving a nonlinear optimization model, the embodiment of the present application can use a preset gradient descent algorithm to solve the optimal value of the camera extrinsic parameter. The preset gradient descent algorithm here can adopt, for example, a GN (Gauss-Newton) algorithm or a LM (Levenberg-Marquardt) algorithm, etc. The specific solution algorithm to be adopted can be flexibly selected by technicians in this field according to actual needs, and no specific limitation is made here.

[0080] In one embodiment of the present application, the external parameters to be corrected include an initial roll angle and an initial pitch angle, the corrected external parameters include a corrected roll angle and a corrected pitch angle, and using the corrected external parameters to correct the external parameters to be corrected of the vehicle-mounted camera includes: replacing the initial roll angle with the corrected roll angle, and replacing the initial pitch angle with the corrected pitch angle.

[0081] As mentioned above, the ground height residual constraint in the embodiment of the present application is mainly used to constrain the optimization of the roll angle and the pitch angle. Therefore, the external parameters to be corrected in the embodiment of the present application refer to the initial roll angle and the initial pitch angle obtained after the initial calibration. After obtaining the corrected roll angle and the corrected pitch angle, the initial roll angle can be directly replaced by the corrected roll angle, and the initial pitch angle can be replaced by the corrected pitch angle. Other external parameters such as translation and yaw angle can remain unchanged, that is, the initial calibrated translation and yaw angle are still used.

[0082] In order to facilitate the understanding of the various embodiments of the present application, the construction process of the nonlinear optimization model of the embodiment of the present application is further listed here in combination with the specific algorithm:

[0083] 1) State quantities to be optimized: roll angle and pitch angle of the camera extrinsic parameters;

[0084] 2) Observation: ground point height Z in vehicle coordinate system obs =0, which can be regarded as a priori hypothesis value;

[0085] 3) Estimation: Z-axis coordinate value Zcar of the road pixel in the vehicle coordinate system

[0086] 4) Ground point height residual constraint:

[0087] The residual term for a single data point:

[0088] f i=e i = Estimator - Observed = Z car -Z obs , (3)

[0089]

[0090] In the above formula, r:3 means taking the value of the third row of the vector, which is Z here. car, .

[0091] Overall optimization objective function:

[0092]

[0093] 5) Initial external parameter constraints:

[0094] Residual term:

[0095]

[0096] Overall optimization objective function:

[0097]

[0098] In the above formula, T is the state quantity to be optimized, that is, the camera external parameter, which includes rotation and translation, T ini is the initial external parameter, ∑ -1 is the information weight matrix, ∑ is the camera extrinsic covariance matrix, different variables correspond to different information weights, log(·) v Represents a logarithmic mapping. For example, the Lie group SE3 can be mapped to the Lie algebra se3, which is convenient for optimization and solution.

[0099] It should be noted that the extrinsic parameter correction process of the vehicle-mounted camera of the present application can be carried out in real time, that is, the extrinsic parameters of the vehicle-mounted camera can be corrected once each frame of road image is acquired. In actual application, since the established nonlinear optimization model is relatively simple, the optimization variables are relatively small, and the initial extrinsic parameters are set as prior constraints, the convergence speed of the nonlinear optimization model is very fast, thereby ensuring the real-time correction of the extrinsic parameters of the vehicle-mounted camera.

[0100] The present application also provides an external parameter calibration device 200 for a vehicle-mounted camera. Figure 2 As shown, a schematic diagram of the structure of an external parameter calibration device for a vehicle-mounted camera in an embodiment of the present application is provided, wherein the device 200 includes: an acquisition unit 210, a construction unit 220, a determination unit 230, and a calibration unit 240, wherein:

[0101] The acquisition unit 210 is used to acquire a road image captured by a vehicle-mounted camera, and recognize the road image using a preset road recognition model to obtain a road recognition result;

[0102] A construction unit 220, configured to construct an external parameter correction constraint condition according to the road surface recognition result, and to construct a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system;

[0103] A determination unit 230, configured to determine a correction extrinsic parameter of the vehicle-mounted camera using the nonlinear optimization model;

[0104] The correction unit 240 is used to obtain the extrinsic parameters to be corrected of the vehicle-mounted camera, and use the correction extrinsic parameters to correct the extrinsic parameters to be corrected of the vehicle-mounted camera.

[0105] In one embodiment of the present application, the predefined coordinate system includes a vehicle body coordinate system, the external parameter correction constraint includes a ground height residual constraint, and the construction unit 220 is specifically used to: determine an estimate of the ground height residual constraint based on the road surface recognition result; determine an observed value of the ground height residual constraint based on a preset ground height value in the vehicle body coordinate system; determine the ground height residual constraint based on the difference between the estimate of the ground height residual constraint and the observed value of the ground height residual constraint.

[0106] In one embodiment of the present application, the predefined coordinate system also includes a camera coordinate system, the road surface recognition result includes the coordinates of road surface pixel points in the camera coordinate system, and the construction unit 220 is specifically used to: convert the coordinates of the road surface pixel points in the camera coordinate system to the vehicle body coordinate system to obtain the coordinates of the road surface pixel points in the vehicle body coordinate system, the coordinates of the road surface pixel points in the vehicle body coordinate system include Z-axis coordinate values ​​in a direction perpendicular to the ground; and use the Z-axis coordinate value as an estimate of the ground height residual constraint condition.

[0107] In one embodiment of the present application, the extrinsic parameter correction constraint also includes an initial extrinsic parameter constraint, and the initial extrinsic parameter constraint includes multiple initial extrinsic parameters and the weights corresponding to each initial extrinsic parameter. The initial extrinsic parameter constraint is determined in the following manner: obtaining multiple initial extrinsic parameters of the vehicle-mounted camera; and determining the weight of each initial extrinsic parameter according to the type of each initial extrinsic parameter.

[0108] In one embodiment of the present application, the external parameters include translation, yaw angle, roll angle and pitch angle, and the initial external parameter constraint condition is determined in the following manner: setting a first weight for the translation and the yaw angle; setting a second weight for the roll angle and the pitch angle; wherein the first weight is greater than the second weight.

[0109] In one embodiment of the present application, the determination unit 230 is specifically used to: solve the nonlinear optimization model using a preset gradient descent algorithm to obtain the correction extrinsic parameters of the vehicle-mounted camera.

[0110] In one embodiment of the present application, the external parameters to be corrected include an initial roll angle and an initial pitch angle, the correction external parameters include a corrected roll angle and a corrected pitch angle, and the correction unit 240 is specifically used to: replace the initial roll angle with the corrected roll angle, and replace the initial pitch angle with the corrected pitch angle.

[0111] It can be understood that the extrinsic parameter correction device of the above-mentioned vehicle-mounted camera can implement the various steps of the extrinsic parameter correction method of the vehicle-mounted camera provided in the aforementioned embodiment. The relevant explanations on the extrinsic parameter correction method of the vehicle-mounted camera are applicable to the extrinsic parameter correction device of the vehicle-mounted camera and will not be repeated here.

[0112] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0113] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0114] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0115] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming an external parameter correction device for the vehicle-mounted camera at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0116] Acquire a road image captured by a vehicle-mounted camera, and use a preset road surface recognition model to recognize the road image to obtain a road surface recognition result;

[0117] Constructing an external parameter correction constraint condition according to the road surface recognition result, and constructing a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system;

[0118] Determining the correction extrinsic parameters of the vehicle-mounted camera using the nonlinear optimization model;

[0119] The extrinsic parameters to be corrected of the vehicle-mounted camera are obtained, and the corrected extrinsic parameters are used to correct the extrinsic parameters to be corrected of the vehicle-mounted camera.

[0120] The above application Figure 1 The method performed by the external parameter correction device of the vehicle-mounted camera disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or an instruction in the form of software. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0121] The electronic device may also perform Figure 1 The method for executing the external parameter correction device of the vehicle-mounted camera in the vehicle-mounted camera is realized Figure 1 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0122] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 1 The method performed by the external parameter correction device of the vehicle-mounted camera in the embodiment shown is specifically used to perform:

[0123] Acquire a road image captured by a vehicle-mounted camera, and use a preset road surface recognition model to recognize the road image to obtain a road surface recognition result;

[0124] Constructing an external parameter correction constraint condition according to the road surface recognition result, and constructing a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system;

[0125] Determining the correction extrinsic parameters of the vehicle-mounted camera using the nonlinear optimization model;

[0126] The extrinsic parameters to be corrected of the vehicle-mounted camera are obtained, and the corrected extrinsic parameters are used to correct the extrinsic parameters to be corrected of the vehicle-mounted camera.

[0127] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0128] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0129] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0130] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0131] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0132] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0134] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0135] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0136] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A method for extrinsic calibration of a vehicle-mounted camera, wherein: The method comprises: Acquire a road image captured by a vehicle-mounted camera, and use a preset road surface recognition model to recognize the road image to obtain a road surface recognition result; Constructing an external parameter correction constraint condition according to the road surface recognition result, and constructing a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system; Determining the correction extrinsic parameters of the vehicle-mounted camera using the nonlinear optimization model; Acquire the to-be-corrected extrinsic parameters of the vehicle-mounted camera, and correct the to-be-corrected extrinsic parameters of the vehicle-mounted camera using the corrected extrinsic parameters; The predefined coordinate system includes a vehicle body coordinate system, the external parameter correction constraint condition includes a ground height residual constraint condition, and the external parameter correction constraint condition constructed according to the road surface recognition result includes: Determining an estimate of the ground height residual constraint condition according to the road surface recognition result; Determine the observed value of the ground height residual constraint condition according to the preset ground height value in the vehicle body coordinate system; Determine the ground height residual constraint condition according to the difference between the estimated amount of the ground height residual constraint condition and the observed amount of the ground height residual constraint condition; The predefined coordinate system also includes a camera coordinate system, the road surface recognition result includes coordinates of road surface pixel points in the camera coordinate system, and the estimator of the ground height residual constraint condition determined according to the road surface recognition result includes: The coordinates of the road pixel points in the camera coordinate system are converted to the vehicle coordinate system to obtain the coordinates of the road pixel points in the vehicle coordinate system, wherein the coordinates of the road pixel points in the vehicle coordinate system include a Z-axis coordinate value in a direction perpendicular to the ground; The Z-axis coordinate value is used as an estimate of the ground height residual constraint condition.

2. The method of claim 1, wherein: The external parameter correction constraint condition also includes an initial external parameter constraint condition, which includes multiple initial external parameters and weights corresponding to each initial external parameter. The initial external parameter constraint condition is determined in the following manner: Acquire multiple initial external parameters of the vehicle-mounted camera; According to the type of each initial external parameter, the weight of each initial external parameter is determined.

3. The method of claim 2, wherein: The external parameters include translation, yaw angle, roll angle and pitch angle. The weight of each initial external parameter is determined according to the type of each initial external parameter, including: Setting a first weight for the translation amount and the yaw angle; Setting a second weight for the roll angle and the pitch angle; The first weight is greater than the second weight.

4. The method of claim 1, wherein: The method of determining the correction extrinsic parameters of the vehicle-mounted camera by using the nonlinear optimization model includes: The nonlinear optimization model is solved using a preset gradient descent algorithm to obtain correction extrinsic parameters of the vehicle-mounted camera.

5. The method of claim 1, wherein: The extrinsic parameters to be corrected include an initial roll angle and an initial pitch angle, the corrected extrinsic parameters include a corrected roll angle and a corrected pitch angle, and the correcting the extrinsic parameters to be corrected of the vehicle-mounted camera using the corrected extrinsic parameters includes: The initial roll angle is replaced by the corrected roll angle, and the initial pitch angle is replaced by the corrected pitch angle.

6. An external parameter calibration device for a vehicle-mounted camera, wherein: The device comprises: An acquisition unit is used to acquire a road image captured by a vehicle-mounted camera, and recognize the road image using a preset road recognition model to obtain a road recognition result; A construction unit, configured to construct an external parameter correction constraint condition according to the road surface recognition result, and to construct a nonlinear optimization model according to the external parameter correction constraint condition and a predefined coordinate system; A determination unit, configured to determine a correction extrinsic parameter of the vehicle-mounted camera using the nonlinear optimization model; A correction unit, used for obtaining the extrinsic parameters to be corrected of the vehicle-mounted camera, and correcting the extrinsic parameters to be corrected of the vehicle-mounted camera using the correction extrinsic parameters; The predefined coordinate system includes a vehicle body coordinate system, the external parameter correction constraint condition includes a ground height residual constraint condition, and the construction unit is specifically used for: Determining an estimate of the ground height residual constraint condition according to the road surface recognition result; Determine the observed value of the ground height residual constraint condition according to the preset ground height value in the vehicle body coordinate system; Determine the ground height residual constraint condition according to the difference between the estimated amount of the ground height residual constraint condition and the observed amount of the ground height residual constraint condition; The predefined coordinate system also includes a camera coordinate system, the road surface recognition result includes the coordinates of road surface pixel points in the camera coordinate system, and the construction unit is specifically used for: The coordinates of the road pixel points in the camera coordinate system are converted to the vehicle coordinate system to obtain the coordinates of the road pixel points in the vehicle coordinate system, wherein the coordinates of the road pixel points in the vehicle coordinate system include a Z-axis coordinate value in a direction perpendicular to the ground; The Z-axis coordinate value is used as an estimate of the ground height residual constraint condition.

7. An electronic device comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 5.

8. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 5.

Images