A vehicle-mounted camera external parameter calibration method and device, electronic equipment and storage medium

Abstract

The application relates to the technical field of camera calibration, and discloses a vehicle-mounted camera external parameter calibration method and device, electronic equipment and a storage medium. According to the method, pose data recorded by a wheel speed odometer and a camera is acquired, target data in which a current frame is a camera pose and two frames before and after the current frame are wheel speed odometer poses is screened from the pose data; based on the target data, the pose of the wheel speed odometer is interpolated into the pose of the camera at the moment, and the frame change amount of the camera and the wheel speed odometer is calculated; the roll angle and the pitch angle of rotation are estimated; data without rotational change is removed; the camera translation is projected onto a plane with the Z axis as the normal line to obtain a global optimization value of camera external parameter calibration; and the camera external parameter is calibrated by using the global optimization value, so that the calibration of the camera external parameter is realized without depending on any special hardware and map prior landmark information.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera calibration, and particularly relates to a vehicle-mounted camera extrinsic parameter calibration method and device, electronic equipment and a storage medium. BACKGROUND

[0002] In the field of intelligent driving, visual sensors have become the basic sensors for vehicles to perceive and obtain environmental information such as surrounding obstacles and road signs. Camera calibration is the key to the accurate perception of surrounding environmental distance information and positioning by vehicles. Camera calibration generally includes camera intrinsic parameters and extrinsic parameters. Camera intrinsic parameters are inherent parameters of the camera, including focal length, optical center and distortion parameters; camera extrinsic parameters are the relative position relationship between the camera and the mounting platform. Although there are many calibration methods for camera extrinsic parameters in the prior art, these calibration methods need to rely on special hardware and map prior information. Therefore, whether the camera extrinsic parameters can be calibrated without relying on special hardware and map prior information has not been solved in the field. SUMMARY

[0003] In view of the above-mentioned shortcomings of the prior art, the present application discloses a vehicle-mounted camera extrinsic parameter calibration method, device, electronic equipment and storage medium, which solves the technical problem of how to calibrate the camera extrinsic parameters without relying on special hardware and map prior information.

[0004] According to an aspect of the embodiments of the present application, a vehicle-mounted camera extrinsic parameter calibration method is provided, which comprises: acquiring pose data recorded by a wheel speed odometer and a camera, wherein the frequency of the wheel speed odometer is different from that of the camera, and the time stamps are kept aligned; selecting target data from the pose data, wherein the current frame is the pose of the camera, and the previous and next two frames are the poses of the wheel speed odometer; based on the target data, interpolating the pose of the wheel speed odometer into the pose of the camera at the moment, and calculating the inter-frame change amount of the camera and the wheel speed odometer; estimating the roll angle and the pitch angle of rotation; removing data without rotational change; projecting the translation of the camera to a plane with the Z-axis as the normal to obtain a global optimization value of the camera extrinsic parameter calibration; and calibrating the camera extrinsic parameter using the global optimization value.

[0005] Preferably, in the vehicle-mounted camera extrinsic parameter calibration method, the pose data includes the rotational transformation and the translation vector of the camera and the wheel speed odometer relative to the map road marker expressed by a quaternion.

[0006] Preferably, in the vehicle-mounted camera extrinsic parameter calibration method, after acquiring the pose data recorded by the wheel speed odometer and the camera, the method further comprises: detecting whether the pose data is null; and if yes, outputting error information.

[0007] Preferably, in the vehicle-mounted camera extrinsic parameter calibration method, the step of interpolating the pose of the wheel speed odometer at the time of the camera into the pose of the camera at the time based on the target data and calculating the inter-frame change amount of the camera and the wheel speed odometer comprises: defining the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel speed odometer, and the relative position of the camera and the vehicle, respectively, wherein the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel speed odometer, and the relative position of the camera and the vehicle satisfy a hand-eye calibration formula; determining the relationship between the relative position of the camera and the vehicle and the map road marker based on the hand-eye calibration formula; interpolating the pose of the wheel speed odometer at the time of the camera into the pose data of the camera; and determining the inter-frame change amount of the camera and the wheel speed odometer based on the relationship between the relative position of the camera and the vehicle and the map road marker and the interpolated pose data of the camera.

[0008] Preferably, in the vehicle-mounted camera extrinsic parameter calibration method, the step of estimating the roll angle and the pitch angle of the rotation comprises: determining the residual of the relative position of the camera and the vehicle; applying disturbance to the residual to establish a least squares equation of the relative position of the camera and the vehicle and the residual; under planar motion, the Z direction in the extrinsic parameter translation of the camera is unobservable, the relative position of the camera and the vehicle is decomposed into Z, Y, and X axes to obtain the relationship of the X direction roll angle and the Y direction pitch angle in the extrinsic parameter rotation of the camera with respect to the relative position of the camera and the vehicle; and based on the inter-frame motion estimation error constraint of the wheel speed odometer and the camera, the roll angle and the pitch angle under the optimal relative position of the camera and the vehicle are solved.

[0009] Preferably, in the vehicle-mounted camera extrinsic parameter calibration method, the image acquisition frequency of the camera is 30 HZ, and the data acquisition frequency of the wheel speed odometer is 100 HZ.

[0010] According to another aspect of the embodiments of the present disclosure, a vehicle-mounted camera extrinsic parameter calibration device is provided, which comprises: a data acquisition module configured to acquire pose data recorded by a wheel speed odometer and a camera, wherein the frequencies of the wheel speed odometer and the camera are different and the time stamps are kept aligned; a data screening module configured to screen target data in which the current frame is the pose of the camera, and the previous and next two frames are the poses of the wheel speed odometer, from the pose data; a calculation module configured to interpolate the pose of the wheel speed odometer at the time of the camera into the pose of the camera at the time based on the target data, and calculate the inter-frame change amount of the camera and the wheel speed odometer; an angle estimation module configured to estimate the roll angle and the pitch angle of the rotation; a data filtering module configured to remove data without rotational change; a translation estimation module configured to project the translation of the camera to a plane with the Z axis as the normal to obtain a global optimization value of the camera extrinsic parameter calibration; and an extrinsic parameter calibration module configured to calibrate the extrinsic parameter of the camera using the global optimization value.

[0011] According to a further aspect of the embodiments of the present disclosure, an electronic device is provided, comprising a processor and a memory; the memory is configured to store a computer program, and the processor is configured to execute the computer program stored in the memory, so that the electronic device performs the above method.

[0012] According to a further aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, which stores computer readable instructions, when the computer readable instructions are executed by a processor of a computer, the computer performs the above method.

[0013] The present disclosure has the beneficial effect that: since the vehicle-mounted camera extrinsic parameter calibration method obtains pose data recorded by a wheel speed odometer and a camera, wherein the frequency of the wheel speed odometer is different from that of the camera and the time stamps are kept aligned; from the pose data, target data is screened in which a current frame is a camera pose, and two frames before and after are wheel speed odometer poses; based on the target data, the pose of the wheel speed odometer is interpolated as the pose at the time of the camera, and the inter-frame change amount of the camera and the wheel speed odometer is calculated, the roll angle and the pitch angle of the rotation are estimated, the data without rotational change is removed, the camera translation is projected to a plane with the Z axis as the normal to obtain a global optimization value of the camera extrinsic parameter calibration, and the camera extrinsic parameter is calibrated using the global optimization value, thus the present disclosure realizes the calibration of the camera extrinsic parameter without relying on any special hardware and map prior landmark information. BRIEF DESCRIPTION OF DRAWINGS

[0014] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application. It is clear that the drawings described below are only some embodiments of the present application, and those skilled in the art can obtain other drawings from these drawings without creative labor. In the drawings:

[0015] Figure 1 is a flowchart of a vehicle-mounted camera extrinsic parameter calibration method provided by an embodiment of the present disclosure;

[0016] Figure 2 is a relative position relationship diagram of a camera and a wheel speed odometer provided by an embodiment of the present disclosure;

[0017] Figure 3 is a structural schematic diagram of a vehicle-mounted camera extrinsic parameter calibration device provided by an embodiment of the present disclosure;

[0018] Figure 4 is a structural schematic diagram of an electronic device in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0019] Following make the specific concrete example explain the implementation of the present application, the person skilled in the art can be easily understood from the present application disclosed by the content of the advantages and efficacy of the present application. The present application can also be implemented or applied by another different specific implementation, the details in the specification can also be based on different views and applications, without departing from the spirit of the present application, various modifications or changes. It should be noted that the following examples and sub-samples in the example can be combined with each other without conflict.

[0020] It should be noted that the drawings provided in the following examples only illustrate the basic concept of the present application in a schematic manner, and only the components related to the present application are shown in the drawings, not the number, shape and size of the components when actually implemented. The actual implementation of each component type, quantity and proportion can be a random change, and the component layout type can be more complex.

[0021] In the following description, a large number of details are discussed to provide a more thorough explanation of the embodiments of the application, however, it is obvious to those skilled in the art that the embodiments of the application can be implemented without these specific details, and in other embodiments, the known structures and devices are shown in the form of block diagrams rather than in detail, to avoid making the embodiments of the application difficult to understand.

[0022] The terms "first", "second", and the like in the description and claims of the present disclosure and the above drawings are used to distinguish similar objects, and do not necessarily have to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances, so that the embodiments of the present disclosure described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion.

[0023] Please refer to Figure 1 A flowchart of a vehicle-mounted camera extrinsic calibration method provided by the present disclosure, in practical applications, the vehicle-mounted camera extrinsic calibration method can be executed by a camera, a vehicle or an electronic device in communication with the vehicle, etc.

[0024] As Figure 1 The vehicle-mounted camera extrinsic calibration method comprises the following steps:

[0025] S10, acquiring the pose data recorded by the wheel speed odometer and the camera, wherein the frequency of the wheel speed odometer is different from that of the camera and the time stamps are kept aligned;

[0026] S20, screening the target data from the pose data, wherein the current frame is the camera pose, and the previous and next two frames are the wheel speed odometer poses;

[0027] S30, based on the target data, interpolating the pose of the wheel speed odometer into the pose of the camera at the moment of the camera, and calculating the inter-frame change amount of the camera and the wheel speed odometer;

[0028] S40, estimating the roll angle and the pitch angle of the rotation;

[0029] S50, removing the data without rotation change;

[0030] S60, projecting the camera translation into a plane with the Z axis as the normal to obtain a global optimization value of camera extrinsic parameter calibration;

[0031] S70, calibrating the camera extrinsic parameter using the global optimization value.

[0032] Specifically, the wheel speed odometer and the camera are both devices provided by the vehicle platform. The present disclosure is applicable to the automatic calibration of the vehicle and the multi-camera in the intelligent driving technology, and does not depend on any special hardware and map prior landmark information. Since the pose data recorded by the wheel speed odometer and the camera are obtained, wherein the frequencies of the wheel speed odometer and the camera are different and the time stamps are kept aligned; the target data in which the current frame is the camera pose and the previous and next frames are the wheel speed odometer poses are selected from the pose data; based on the target data, the pose of the wheel speed odometer is interpolated into the pose of the camera at the moment of the camera, and the inter-frame change amount of the camera and the wheel speed odometer is calculated; the roll angle and the pitch angle of the rotation are estimated; the data without rotation change is removed; the camera translation is projected into a plane with the Z axis as the normal to obtain a global optimization value of camera extrinsic parameter calibration; the camera extrinsic parameter is calibrated using the global optimization value. Therefore, the present disclosure realizes the calibration of the camera extrinsic parameter without depending on any special hardware and map prior landmark information, and can reduce the influence of the extrinsic parameter error on the positioning accuracy.

[0033] In the above step S10, the frequency of the wheel speed odometer is higher than that of the camera, so as to interpolate the pose frame recorded by the wheel speed odometer into the pose frame of the camera. For example, the image acquisition frequency of the camera is 30HZ, and the data acquisition frequency of the wheel speed odometer is 100HZ.

[0034] The wheel speed odometer is one of the simplest forms of vehicle self-positioning, and is widely used in many steering robots and new energy vehicles. In the present embodiment, when the vehicle is in a uniform speed driving state, the wheel speed odometer and the camera are controlled to record the pose data of the vehicle in the driving track.

[0035] Specifically, the recorded pose data includes the rotation transformation and the translation vector of the camera and the wheel speed odometer relative to the map landmark point expressed by a quaternion. The motion equation of the vehicle recorded by the wheel speed odometer is:

[0036]

[0037] where θ and p are yaw angle and position in the plane of motion, R(θ) is a rotation in the plane of motion, R(θ) 2，2 denotes a 2x2 rotation matrix, 0 1，2 denotes a 1x2 matrix, 0 2，1 denotes a 2x1 matrix, v is linear velocity (including forward and lateral linear velocity), ω is angular velocity, where the subscript denotes the dimension of the vector or matrix.

[0038] Due to environmental interference and other factors, the pose data collected by the wheel speed odometer and the camera may have abnormal values. Therefore, before processing the obtained pose data, it is necessary to detect the pose data to obtain effective pose data.

[0039] For example, after obtaining the pose data recorded by the wheel speed odometer and the camera, the following steps are further included: detecting whether the pose data is null; if so, outputting error information; if not, entering the next step, or outputting the corresponding pose data and returning to detecting whether the pose data is null. The embodiment excludes some erroneous data in the record by performing null detection on the obtained pose data.

[0040] In the above step S20, since the frequency of the wheel speed odometer is higher than that of the camera, in the same time, the wheel speed odometer obtains more pose frames than the camera. If the pose data obtained by the wheel speed odometer and the camera is arranged in time sequence, then the target data in which the current frame is the camera pose and the previous and next two frames are the wheel speed odometer pose can be screened out. For example, the pose frame sequence collected by the camera is {M1, M2, M3}, and the pose frame sequence collected by the wheel speed odometer is {L1, L2, L3, L4, L5, L6, L7, L8, L9}. Since the time stamps of the two are aligned, the pose data collected by the camera and the wheel speed odometer in time sequence is: L1→L2→L3(M1)→L4→L5→L6(M2)→L7→L8→L9(M3). If the time of the camera is taken as the reference, then the target data includes: L2→L3(M1)→L4, L6(M2)→L7.

[0041] In some embodiments, the step S30 of interpolating the pose of the wheel speed odometry into the pose of the camera at the camera time based on the target data and calculating the inter-frame change amount of the camera and the wheel speed odometry comprises: defining the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel speed odometry and the relative position of the camera and the vehicle, respectively, wherein the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel speed odometry and the relative position of the camera and the vehicle satisfy the hand-eye calibration formula; determining the relationship between the relative position of the camera and the vehicle and the map road marker based on the hand-eye calibration formula; interpolating the pose of the wheel speed odometry at the camera time into the pose data of the camera; and determining the inter-frame change amount of the camera and the wheel speed odometry based on the relationship between the relative position of the camera and the vehicle and the map road marker and the interpolated pose data of the camera.

[0042] Please refer to Figure 2 , it is assumed that the inter-frame motion rotation of the camera is defined as , the inter-frame motion translation is defined as , the inter-frame motion rotation recorded by the vehicle wheel speed odometry is defined as , the inter-frame motion translation is defined as , the posture rotation of the vehicle relative to the camera is defined as , and the posture translation is defined as O p C . According to the hand-eye calibration formula AX = XB, it is obtained that wherein o i+1 and o i represent two adjacent frames, and i is a non-negative integer.

[0043] Therefore, the relationship between the road marker and o can be expressed as:

[0044]

[0045] wherein R is the rotation transformation of the camera and the road marker, and I3 represents a unit matrix.

[0046] wherein the three-dimensional rotation can be represented by a three-dimensional matrix (SO(3)), a rotation vector (so(3)), a quaternion and Euler angles, and the first three representations of rotation are usually used in a SLAM (Simultaneous Localization And Mapping) system due to the gimbal lock problem of Euler angles. The rotation matrix around the fixed axis can be expressed as Euler angles:

[0047] R OC = R z (γ)*R y (β)*R x (α);

[0048] wherein Rz (γ) is a γ angle decomposed around the z axis, R y (β) is a β angle decomposed around the y axis; R x (α) is a β angle decomposed around the y axis.

[0049] Then, it can be expressed by using quaternions as:

[0050]

[0051] where q z (γ), q y (β), q x (α) are quaternions of decomposed angles along x, y, z axes respectively.

[0052] The current frame is the camera pose, and the front and rear two frames are the data of the wheel speed odometer. The pose data of the odometer at the camera time is interpolated as the pose data sequence of the camera based on the pose of the camera, so as to calculate the inter-frame change amount of the camera and the wheel speed odometer and

[0053] Then, in the step S40, the roll angle and the pitch angle of the rotation are estimated, including: determining the residual of the relative position of the camera and the vehicle; applying disturbance to the residual to establish a least square equation of the relative position of the camera and the vehicle and the residual; under the plane motion, the Z direction in the external parameter translation of the camera is unobservable, the relative position of the camera and the vehicle is decomposed into Z, Y and X axes, the X direction roll angle and the Y direction pitch angle in the external parameter rotation of the camera are obtained about the relative position of the camera and the vehicle; based on the inter-frame motion estimation error constraint of the wheel speed odometer and the camera, the roll angle and the pitch angle under the optimal relative position of the camera and the vehicle are solved.

[0054] Specifically, it is assumed that the vehicle moves on a plane, the external parameter translation of the Z direction of the camera is unobservable, the x direction roll angle (i.e. roll angle) and the y direction pitch angle (i.e. pitch angle) in the external parameter rotation are estimated, and the rotation external parameter is decomposed into Z, Y and X axes, then:

[0055]

[0056] q z (γ) = [0 0 sin(γ / 2) cos(γ / 2)] T ;

[0057] q y (β) = [0 sin(β / 2) 0 cos(β / 2)] T ;

[0058] q x(α) = [sin(α / 2) 0 0 cos(α / 2)] T ;

[0059] wherein, The residual of q

[0060] Thus, we have wherein, the wheel odometry rotation is around the Z axis, so and q z (γ) can be exchanged:

[0061]

[0062] Since is an orthogonal matrix, it will not affect the size of the error. For Apply perturbation to establish and η i The least square equation of q

[0063]

[0064] wherein, The least square optimal solution is represented by q yx The calibration of q

[0065]

[0066] wherein, M represents the inter-frame motion disturbance set of the camera and the vehicle odometry, SVD decomposition is performed on M'M, the eigenvectors corresponding to the smallest two eigenvalues are defined as t1 and t2, and the final solution can be obtained by linear combination:

[0067] Then, the coefficients a and b of the linear combination can be solved by constraining ||q yx ||2=1, to obtain the roll angle and pitch angle of q .

[0068] In the above step S60, the translation of the camera is projected to the plane with the normal of R yx The third row (Z axis) is the normal, thereby outputting the dimension of the residual, optimizing the rotation, translation and scale of the external parameter. Thus, the calibration of the camera external parameter is realized.

[0069] All the above optional technical solutions can be combined to form optional embodiments of the present application, which will not be repeated here.

[0070] The following is an embodiment of the device of the present disclosure, which can be used to execute the method embodiment of the present disclosure. For details not disclosed in the device embodiment of the present disclosure, please refer to the method embodiment of the present disclosure.

[0071] Figure 3 is a schematic diagram of a vehicle-mounted camera extrinsic parameter calibration device provided by an embodiment of the present disclosure. As shown in Figure 3 , the vehicle-mounted camera extrinsic parameter calibration device comprises:

[0072] The data acquisition module 301 is configured to acquire pose data recorded by a wheel speed odometer and a camera, wherein the frequency of the wheel speed odometer is different from that of the camera and the timestamps are kept aligned.

[0073] The data screening module 302 is configured to screen target data from the pose data, wherein the current frame is the pose of the camera, and the two frames before and after are the poses of the wheel speed odometer.

[0074] The calculation module 303 is configured to interpolate the pose of the wheel speed odometer into the pose at the time of the camera based on the target data, with the pose of the camera as the reference, and calculate the inter-frame change amount of the camera and the wheel speed odometer.

[0075] The angle estimation module 304 is configured to estimate the roll angle and the pitch angle of rotation.

[0076] The data filtering module 305 is configured to remove data without rotational changes.

[0077] The translation estimation module 306 is configured to project the camera translation to a plane with the Z-axis as the normal to obtain a globally optimized value of camera extrinsic parameter calibration.

[0078] The extrinsic parameter calibration module 307 is configured to calibrate the camera extrinsic parameter using the globally optimized value.

[0079] According to the technical scheme provided by the embodiment of the present disclosure, by acquiring pose data recorded by a wheel speed odometer and a camera, wherein the frequency of the wheel speed odometer is different from that of the camera and the timestamps are kept aligned; screening target data from the pose data, wherein the current frame is the pose of the camera, and the two frames before and after are the poses of the wheel speed odometer; interpolating the pose of the wheel speed odometer into the pose at the time of the camera based on the target data, with the pose of the camera as the reference, and calculating the inter-frame change amount of the camera and the wheel speed odometer; estimating the roll angle and the pitch angle of rotation; removing data without rotational changes; projecting the camera translation to a plane with the Z-axis as the normal to obtain a globally optimized value of camera extrinsic parameter calibration; and calibrating the camera extrinsic parameter using the globally optimized value, therefore, the present disclosure realizes the calibration of the camera extrinsic parameter without relying on any special hardware and map prior landmark information, and can reduce the influence of extrinsic parameter error on positioning accuracy.

[0080] In some embodiments, the pose data includes a rotation transformation and a translation vector of the camera and the wheel odometry relative to the map landmark points expressed by quaternions.

[0081] In some embodiments, the vehicle-mounted camera extrinsic parameter calibration device further comprises:

[0082] The data detection module 308 is configured to detect whether the pose data recorded by the wheel odometry and the camera is empty after the pose data is acquired.

[0083] The data error reporting module 309 is configured to output error information if the pose data is empty.

[0084] In some embodiments, Figure 3 The calculation module 303 is specifically configured to define the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel odometry, and the relative position of the camera and the vehicle, wherein the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel odometry, and the relative position of the camera and the vehicle satisfy a hand-eye calibration formula; determine the relationship between the relative position of the camera and the vehicle and the map landmark points based on the hand-eye calibration formula; interpolate the pose of the wheel odometry at the time of the camera as the pose data of the camera; and determine the inter-frame change amount of the camera and the wheel odometry based on the relationship between the relative position of the camera and the vehicle and the map landmark points and the interpolated pose data of the camera.

[0085] In some embodiments, Figure 3 The angle estimation module 304 is specifically configured to determine the residual error of the relative position of the camera and the vehicle; apply perturbation to the residual error to establish a least squares equation of the relative position of the camera and the vehicle and the residual error; under planar motion, the Z direction in the extrinsic parameter translation of the camera is unobservable, the relative position of the camera and the vehicle is decomposed into Z, Y and X axes to obtain the relationship between the X roll angle and the Y pitch angle in the extrinsic parameter rotation of the camera relative to the relative position of the camera and the vehicle; based on the inter-frame motion estimation error constraint of the wheel odometry and the camera, the roll angle and the pitch angle under the optimal relative position of the camera and the vehicle are solved.

[0086] In some embodiments, the image acquisition frequency of the camera is 30HZ, and the data acquisition frequency of the wheel odometry is 100HZ.

[0087] Please refer to Figure 4 The present disclosure provides a structural schematic diagram of an electronic device, as shown in Figure 4As shown, the electronic device 400 includes a central processing unit (CPU) 401 which can perform various appropriate actions and processes, such as executing the methods in the above-described embodiments, according to a program stored in a read-only memory (ROM) 402 or a program loaded from the storage section 408 into a random access memory (RAM) 403. In the RAM 403, various programs and data required for system operation are also stored. The CPU 401, the ROM 402, and the RAM 403 are connected to each other through a bus 404. An input / output (I / O) interface 405 is also connected to the bus 404.

[0088] Connected to the I / O interface 404 are an input section 406 including a keyboard, a mouse, etc.; an output section 407 including a display such as a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 408 including a hard disk, etc.; and a communication section 409 including a network interface card such as a LAN (Local Area Network) card, a modem, etc. The communication section 409 performs communication processing via a network such as the Internet. A drive 410 is also connected to the I / O interface 404 as necessary. A removable recording medium 411 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is attached to the drive 410 as necessary, so that a computer program read therefrom is installed into the storage section 408 as necessary.

[0089] In particular, according to embodiments of the present application, the processes described above with reference to the flowcharts can be implemented as a computer software program. For example, embodiments of the present application include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing a computer program for executing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via the communication section 409, and / or installed from the removable recording medium 411. When the computer program is executed by the central processing unit (CPU) 401, various functions defined in the system of the present application are performed.

[0090] It should be noted that the computer readable medium shown in the embodiments of the present application can be a computer readable signal medium or a computer readable storage medium or any combination of the two. The computer readable storage medium may, for example, be an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples of the computer readable storage medium can include, but are not limited to, an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, the computer readable signal medium can include a data signal propagating in the baseband or as a carrier wave part of a signal propagating in the carrier wave, in which the computer readable computer program is carried. Such a propagating data signal can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer readable signal medium can also be any computer readable medium other than the computer readable storage medium, which can send, propagate or transmit programs for use by or in connection with an instruction execution system, device or apparatus. The computer program contained on the computer readable medium can be transmitted by any suitable medium, including but not limited to wireless, wired, etc., or any suitable combination thereof.

[0091] The embodiments of the present disclosure also provide a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement any of the methods in the embodiments.

[0092] The computer readable storage medium in the embodiments of the present disclosure can be understood by those skilled in the art that all or part of the steps of the above-mentioned method embodiments can be completed by a computer program related hardware. The aforementioned computer program can be stored in a computer readable storage medium. The program is executed to perform the steps of the above-mentioned method embodiments; and the aforementioned storage medium includes ROM, RAM, magnetic disk or optical disk and various media that can store program codes.

[0093] The electronic device disclosed in the embodiments includes a processor, a memory, a transceiver and a communication interface. The memory and the communication interface are connected with the processor and the transceiver and complete communication between each other. The memory is used to store a computer program, the communication interface is used for communication, and the processor and the transceiver are used to run the computer program, so that the electronic device executes each step of the method as above.

[0094] In the present embodiment, the memory can include a random access memory (RAM) and can also include a non-volatile memory, such as at least one disk memory.

[0095] The processor described above can be a general processor, including a central processing unit (CPU), a graphics processing unit (GPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component.

[0096] The above description and drawings are illustrative of embodiments of the present disclosure and are not intended to be limiting. Other embodiments can include structural, logical, electrical, process, and other changes. Embodiments are merely representative of possible variations. Individual components and functions are optional and the order of operations can vary. Portions and sub-combinations of some embodiments can be included or replaced in or by other embodiments. Also, the phraseology and terminology used herein is for the purpose of description and should not be regarded as limiting. As used in the description of the embodiments and the claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. Similarly, the term "and / or" as used herein refers to any and all possible combinations of one or more of the associated listed items. Additionally, as used in this application, the term "comprises" and variations thereof do not intend to preclude the presence or addition of one or more other items to those stated in the compositions, integers, steps, operations, elements, and / or components, but to affix the existence of those items which the phrase "comprises" refers to. Without more limitations, an element defined by the phrase "comprising... " does not exclude the existence of additional identical elements that are also included in "processes, methods, or apparatuses" that include the element. In this document, each embodiment focuses on the differences from other embodiments, and the same or similar parts between embodiments can be referred to each other. For the methods, products, etc. disclosed in the embodiments, if they correspond to the method part of the embodiments, the relevant parts can be referred to the description of the method part.

[0097] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are realized in hardware or software depends on the specific application and design constraints of the technical solution. The skilled person can use different methods for each specific application to realize the described functions, but such implementation should not be considered beyond the scope of the embodiments of the present disclosure. The skilled person can clearly understand that, for the convenience and brevity of description, the specific working process of the above-described system, device and unit can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0098] In the embodiments disclosed herein, the disclosed methods, products (including but not limited to devices, apparatuses, etc.) can be implemented in other manners. For example, the described device embodiments are merely illustrative. For example, the division of the units is merely logical function division. There can be another division manner for the actual implementation. For example, a plurality of units or components can be combined or integrated into another system, or some of the sub-units can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections can be indirect couplings or communication connections through some interfaces, devices or units, and can be in electrical, mechanical or other forms. The units described as separated components can or can not be physically separated, and can or can not be physical units. Some or all of the units can be implemented in a form of a processing element. In addition, the units described as separate components can be or can not be physical units. They can be located in one position, or distributed on a plurality of network units.

[0099] The flowcharts and block diagrams in the drawings show the possible implementation architectures, functions and operations of the system, method and computer program product according to the embodiments of the present disclosure. In this regard, each block in the flowcharts or block diagrams can represent a module, a program segment or a part of code containing one or more executable instructions for implementing the specified logical function. In some alternative implementations, the functions noted in the blocks can occur in different orders than those noted in the drawings. For example, two consecutive blocks can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. In the descriptions corresponding to the flowcharts and block diagrams in the drawings, the operations or steps corresponding to different blocks can also occur in different orders than those disclosed in the descriptions, and sometimes there is no specific order between different operations or steps. For example, two consecutive operations or steps can actually be executed substantially in parallel, and sometimes they can be executed in reverse order, depending on the functions involved. Each block in the block diagrams and / or flowcharts, and the combination of blocks in the block diagrams and / or flowcharts, can be implemented by a dedicated hardware-based system for implementing the specified functions or actions, or can be implemented by a combination of dedicated hardware and computer instructions.

Claims

1. A method for calibrating the extrinsic parameters of a vehicle-mounted camera, characterized in that, include: Acquire pose data recorded by wheel speed odometer and camera, wherein the wheel speed odometer and camera have different frequencies and their timestamps are aligned; The current frame is selected as the camera pose from the pose data, and the two frames before and after are the target data for the wheel speed odometer pose. Based on the target data, with the camera pose as the reference, the wheel speed odometer pose is interpolated to the camera pose at any given time, and the inter-frame change between the camera and the wheel speed odometer is calculated. Estimate the roll and pitch angles of rotation; Remove data that has not undergone rotational changes; The camera is translated and projected onto a plane with the Z-axis as the normal, and the global optimized values ​​of the camera extrinsic parameters are obtained. The camera extrinsic parameters are calibrated using the global optimized values.

2. The method for calibrating the extrinsic parameters of a vehicle-mounted camera according to claim 1, characterized in that, The pose data includes rotation and translation vectors of the camera and wheel speed odometer relative to map landmarks, expressed in quaternions.

3. The method for calibrating the extrinsic parameters of a vehicle-mounted camera according to claim 2, characterized in that, After acquiring the pose data recorded by the wheel speed odometer and camera, the following is also included: Detect whether the pose data is empty; If so, output an error message.

4. The method for calibrating the extrinsic parameters of a vehicle-mounted camera according to claim 2, characterized in that, Based on the target data, and using the camera pose as a reference, the wheel speed odometer pose is interpolated to the camera pose at that moment, and the inter-frame change between the camera and the wheel speed odometer is calculated, including: Define the following separately: the inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel speed odometer, and the relative position of the camera and the vehicle. The inter-frame motion of the camera, the inter-frame motion recorded by the vehicle wheel speed odometer, and the relative position of the camera and the vehicle satisfy the hand-eye calibration formula. Based on the aforementioned hand-eye calibration formula, the relationship between the relative positions of the camera and the vehicle and the map landmarks is determined. The wheel speed odometer pose at the camera moment is interpolated to the camera pose data; Based on the relationship between the relative position of the camera and the vehicle and the map landmarks, and the interpolated camera pose data, the inter-frame variation between the camera and the wheel speed odometer is determined.

5. The method for calibrating the extrinsic parameters of a vehicle-mounted camera according to claim 4, characterized in that, Estimate the roll and pitch angles of rotation, including: Determine the residuals of the relative positions of the camera and the vehicle; A perturbation is applied to the residual to establish the least squares equations for the relative positions of the camera and the vehicle and the residual; In planar motion, the Z-direction of the camera's extrinsic translation is not observable. By decomposing the relative position of the camera and the vehicle along the Z, Y, and X axes, we can obtain the relationship between the X-direction roll angle and the Y-direction pitch angle of the camera's extrinsic rotation and the relative position of the camera and the vehicle. Based on the inter-frame motion estimation error constraints of the wheel speed odometer and the camera, the roll angle and pitch angle under the optimal relative position of the camera and the vehicle are obtained.

6. The method for calibrating the extrinsic parameters of a vehicle-mounted camera according to any one of claims 1-5, characterized in that, The camera has an image acquisition frequency of 30Hz, and the wheel speed odometer has a data acquisition frequency of 100Hz.

7. A vehicle-mounted camera extrinsic parameter calibration device, characterized in that, include: The data acquisition module is configured to acquire pose data recorded by the wheel speed odometer and the camera, wherein the wheel speed odometer and the camera have different frequencies and their timestamps are aligned. The data filtering module is configured to filter target data from the pose data, where the current frame is the camera pose and the two frames before and after are the wheel speed odometer poses. The calculation module is configured to, based on the target data and with the camera pose as a reference, interpolate the pose of the wheel speed odometer to the pose of the camera at any given time, and calculate the inter-frame change between the camera and the wheel speed odometer. The angle estimation module is configured to estimate the roll and pitch angles of rotation; The data filtering module is configured to remove data that has not undergone rotational changes; The translation estimation module is configured to translate and project the camera onto a plane with the Z-axis as the normal, thereby obtaining the global optimized values ​​for the camera extrinsic calibration. The extrinsic parameter calibration module is configured to calibrate the camera extrinsic parameters using the global optimized values.

8. The vehicle-mounted camera extrinsic parameter calibration device according to claim 7, characterized in that, Also includes: The data detection module is configured to detect whether the pose data is empty; The data error reporting module, if configured to do so, will output error information.

9. An electronic device, characterized in that, include: Processor and memory; The memory is used to store a computer program, and the processor is used to execute the computer program stored in the memory to cause the electronic device to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, It stores computer-readable instructions that, when executed by the computer's processor, cause the computer to perform the method of any one of claims 1 to 7.

Images