Camera external parameter calculation method and device, electronic equipment and storage medium

Abstract

The invention discloses a camera external parameter calculation method and device, electronic equipment and a storage medium, and the method comprises the steps: building a geometrical relationship between a camera and a lane line according to camera parameters, lane line information and lane information which are obtained in advance; according to the geometrical relationship, determining a unit vector which passes through O and is parallel to the lane line; according to a normal vector of a target plane, the unit vector which passes through the O and is parallel to the lane line and the lane information, coordinates of a point on an intersection line of a plane where the lane line is located and a vertical plane where the lane line passes through the O point in a camera coordinate system are calculated; based on the local coordinate system, external parameters of the camera in the local coordinate system are determined; according to the high-precision map data of the target area, external parameters of the local coordinate system in a world coordinate system are determined; and determining the external parameter of the camera in the world coordinate system according to the external parameter of the local coordinate system in the world coordinate system and the external parameter of the camera in the local coordinate system. According to the invention, preliminary calculation of the external parameters of the camera is realized.

Description

Technical Field

[0001] The present application relates to the technical field of camera extrinsic parameter calculation, and in particular to a camera extrinsic parameter calculation method, device, electronic device, and storage medium. Background Art

[0002] Cameras and other sensors need to be installed on roadside poles for roadside perception. When cameras are installed on roadside poles and high-precision map data is available on the road, accurate camera extrinsics are usually obtained by combining initial values ​​with optimization strategies. Therefore, accurate camera extrinsics depend on good initial values.

[0003] For the initial value of the extrinsic parameter, the method of calculating the initial value using the vanishing point can only estimate two of the three degrees of freedom of rotation in the extrinsic parameter, but cannot estimate the translation in the extrinsic parameter. The reason is that the pixel coordinates of the vanishing point can only obtain the angle between the camera's main optical axis and the parallel line beam, and this information only contains two degrees of freedom. Summary of the invention

[0004] The embodiments of the present application provide a camera extrinsic parameter calculation method, device, electronic device, and storage medium, which can not only estimate all degrees of freedom of rotation and height in the translation vector in the camera extrinsic parameters, but also estimate the remaining two translation quantities in the extrinsic parameters with the help of simple auxiliary means.

[0005] The present application embodiment adopts the following technical solutions:

[0006] In a first aspect, an embodiment of the present application provides a method for calculating camera extrinsic parameters, wherein the method comprises:

[0007] According to the pre-acquired camera parameters, lane line information and lane information, a geometric relationship between the camera and the lane line is established;

[0008] According to the geometric relationship, determine a unit vector passing through O and parallel to the lane line, where O is the center of the camera model after dedistortion;

[0009] According to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are calculated, and the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system;

[0010] Based on the local coordinate system, determine the external parameters of the camera in the local coordinate system;

[0011] Determine the external parameters of the local coordinate system in the world coordinate system based on the high-precision map data of the target area;

[0012] Determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

[0013] In some embodiments, determining the external parameters of the camera in the local coordinate system based on the local coordinate system includes:

[0014] Based on the local coordinate system, a rotation matrix from the local coordinate system to the camera coordinate system and a translation vector from the local coordinate system to the camera coordinate system are calculated;

[0015] Determine the extrinsic parameters of the camera in the local coordinate system according to the rotation matrix from the local coordinate system to the camera coordinate system and the translation vector from the local coordinate system to the camera coordinate system.

[0016] In some embodiments, determining the external parameters of the local coordinate system in the world coordinate system according to the high-precision map data of the target area includes:

[0017] According to the high-precision map (provided plane normal vectors and lane line information), calculate the rotation matrix and translation vector between the world coordinate system and the local coordinate system;

[0018] Determine the translation vector between the local coordinate system and the world coordinate system based on the coordinate position of the pole where the camera is located in the world coordinate system and the corresponding position on the lane line corresponding to the high-precision map;

[0019] According to the rotation matrix between the world coordinate system and the local coordinate system and the translation vector between the local coordinate system and the world coordinate system, the extrinsic parameters of the local coordinate system in the world coordinate system are determined.

[0020] In some embodiments, determining the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system includes:

[0021] Determine the external parameter parameters of the camera in the world coordinate system = the external parameter parameters of the local coordinate system in the world coordinate system * the external parameter parameters of the local coordinate system in the camera coordinate system -1 .

[0022] In some embodiments, determining, based on the geometric relationship, a unit vector passing through O and parallel to the lane line includes:

[0023] According to the camera imaging model, obtain the unit vector extending outward from point O;

[0024] Calculate the normal vector of the target plane according to the unit vector extending outward from point O;

[0025] According to the normal vector of the target plane, a unit vector passing through O and parallel to the lane line is calculated.

[0026] In some embodiments, the coordinates of the points on the intersection of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system, including:

[0027] According to the coordinates of the points on the intersection of the plane where the lane line is located and the perpendicular plane passing through point O in the camera coordinate system, the local coordinate system is established with the first direction as the x-axis, the second direction as the y-axis, and the direction of the cross product of the first direction vector and the second direction as the z-axis. The first direction and the second direction are obtained according to the geometric relationship.

[0028] In some embodiments, a geometric relationship between the camera and the lane line is established based on pre-acquired camera parameters, lane line information, and lane information, including:

[0029] A geometric relationship between the camera and the lane line is established based on pre-acquired camera intrinsic parameters, information of at least three parallel lane lines, and lane line widths, wherein the lane line information includes pixel coordinates located on the lane line.

[0030] In a second aspect, an embodiment of the present application further provides a device for calculating an initial value of a camera extrinsic parameter, wherein the device comprises:

[0031] A preprocessing module, used to establish a geometric relationship between the camera and the lane line according to the pre-acquired camera parameters, lane line information and lane information;

[0032] A first determination module is used to determine a unit vector passing through O and parallel to the lane line according to the geometric relationship, where O is the center of the camera model after dedistortion;

[0033] A calculation module, for calculating the coordinates of a point on the intersection line of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system according to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, wherein the coordinates of the point on the intersection line of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system are used to establish a local coordinate system;

[0034] A second determination module is used to determine the external parameters of the camera in the local coordinate system based on the local coordinate system;

[0035] The third determination module is used for the high-precision map data of the target area to determine the external parameters of the local coordinate system in the world coordinate system;

[0036] The fourth determination module is used to determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

[0037] In a third aspect, an embodiment of the present application further provides an electronic device, comprising: a processor; and a memory arranged to store computer executable instructions, wherein the executable instructions, when executed, cause the processor to perform the above method.

[0038] In a fourth aspect, an embodiment of the present application further provides a computer-readable storage medium, which stores one or more programs. When the one or more programs are executed by an electronic device including multiple application programs, the electronic device executes the above method.

[0039] At least one of the above technical solutions adopted in the embodiments of the present application can achieve the following beneficial effects: according to the pre-acquired camera parameters, lane line information and lane information, a geometric relationship between the camera and the lane line is established. Then, according to the geometric relationship, a unit vector passing through O and parallel to the lane line is determined, and O is the center of the camera model after dedistortion. Then, according to the normal vector of the target plane, the unit vector passing through O and parallel to the lane line, and the lane information, the coordinates of the points on the intersection of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are calculated, and the coordinates of the points on the intersection of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system. Thus, based on the local coordinate system, the external parameter parameters of the camera in the local coordinate system are determined; according to the high-precision map data of the target area, the external parameter parameters of the local coordinate system in the world coordinate system are determined; according to the external parameter parameters of the local coordinate system in the world coordinate system and the external parameter parameters of the camera in the local coordinate system, the external parameter parameters of the camera in the world coordinate system are determined. Through the above method, the initial values ​​of the external parameters of the camera in the road coordinate system and the world coordinate system can be calculated. BRIEF DESCRIPTION OF THE DRAWINGS

[0040] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0041] Figure 1 This is a schematic diagram of an image used to calculate camera extrinsic parameters in an embodiment of the present application;

[0042] Figure 2 Schematic diagram of the process of calculating camera extrinsic parameters in an embodiment of the present application;

[0043] Figure 3This is a schematic diagram of the geometric relationship between the camera and the lane line in the embodiment of the present application;

[0044] Figure 4 This is a schematic diagram of the structure of a camera extrinsic parameter calculation device in an embodiment of the present application;

[0045] Figure 5 This is a schematic diagram of the structure of an electronic device in an embodiment of the present application. DETAILED DESCRIPTION

[0046] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0047] The camera intrinsic parameter describes the correspondence between the point in the camera coordinate system and the pixel coordinate system. Assume that P is a point in the camera coordinate system. The camera moves in three-dimensional space, and the point P in the world coordinate system is recorded as Pw, and the coordinates in the camera coordinate system are recorded as Pc. The position and posture of the camera in the world coordinate system are described by the camera's rotation matrix R and translation vector t.

[0048] When calculating the camera extrinsics, the known quantities that need to be obtained include:

[0049] (1) Camera intrinsic parameters. The camera’s intrinsic parameters are fixed after leaving the factory. If the camera’s factory parameters are known, the camera’s intrinsic parameters can be calculated.

[0050] (2) Three parallel lane lines. Figure 1 As shown, each lane line is described by any two points on the lane line center axis in the image. And it is known that Figure 1 The pixel coordinates corresponding to A, B, C, D, E, and F.

[0051] (3) Figure 1 As shown, the lane widths d1 and d2 can be approximately equal in ordinary road scenes.

[0052] (4) Take the coordinates of the roadside pole in the world coordinate system start_point: (x0, y0, z0), which can be approximated as the contact point between the roadside pole and the ground. Only the position information of this point along the road direction affects the final result.

[0053] (5) High-precision maps correspond to Figure 1 The leftmost lane line CD shown, and any point on any lane on the non-CD lane line here.

[0054] The technical solutions provided by various embodiments of the present application are described in detail below in conjunction with the accompanying drawings.

[0055] The present application embodiment provides a method for calculating camera extrinsic parameters, such as Figure 2 As shown, a schematic flow chart of a method for calculating camera extrinsic parameters in an embodiment of the present application is provided, and the method at least includes the following steps S210 to S260:

[0056] Step S210, establishing a geometric relationship between the camera and the lane line according to the pre-acquired camera parameters, lane line information and lane information.

[0057] According to the actual scenario of installing cameras on roadside poles, camera parameters, lane line information and lane information are acquired in advance. It can be understood that camera parameters mainly refer to camera internal parameters. Lane line information includes multiple parallel lane lines and corresponding lane lines in high-precision map data. Lane information mainly refers to lane width, which is usually equal.

[0058] It should be noted that the camera installed on the roadside pole can be a box camera or a fisheye camera.

[0059] According to the pre-obtained camera parameters, lane line information and lane information, the following can be established: Figure 3 The camera and lane geometry shown.

[0060] Step S220: Determine a unit vector passing through O and parallel to the lane line according to the geometric relationship, where O is the center of the camera model after dedistortion.

[0061] Please continue to refer to Figure 3 ,exist Figure 3 AB, CD, EF are the right, left, and center lane lines respectively. The pixel coordinate information of each point is known. The same symbol is used to represent the same physical point in the image and three-dimensional space. O is the center of the camera pinhole model after dedistortion. I, H, G are the points on the intersection of the plane where the lane line is located and the perpendicular plane passing through point O.

[0062] Based on the geometric relationship, a unit vector passing through O and parallel to the lane line can be determined.

[0063] Step S230, based on the normal vector of the target plane, the unit vector passing through O and parallel to the lane line, and the lane information, calculate the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system. The coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system.

[0064] According to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, the coordinates of the point on the intersection of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system are further calculated, as follows: Figure 3 As shown, the coordinates of I, H, and G in the cam coordinate system can be obtained accordingly.

[0065] Step S240: Determine the external parameters of the camera's local coordinate system based on the local coordinate system.

[0066] The local coordinate system can be obtained based on the coordinates of I, H, and G in the cam coordinate system, and then the external parameters of the camera's local coordinate system can be determined based on the local coordinate system.

[0067] Step S250, determining the external parameters of the local coordinate system in the world coordinate system based on the high-precision map data of the target area.

[0068] The normal vector of the target plane can be obtained through the high-precision map data of the target area.

[0069] According to the normal vector of the target plane, lane line information, and starting point information provided by the high-precision map data of the target area, the external parameter parameters of the local coordinate system in the world coordinate system are determined.

[0070] Step S260, determining the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

[0071] Finally, according to the extrinsic parameters (rotation matrix and translation vector) of the local coordinate system in the world coordinate system, the extrinsic parameters of the camera in the world coordinate system are determined.

[0072] Through the above method, the position of the three lane lines in the image and the lane width can be used to obtain the position of the camera relative to the lane lines, and then the transformation relationship of the lane lines relative to the world coordinate system can be obtained through the high-precision map, and the external parameters of the camera in the world coordinate system can be obtained. It can be used on box cameras and fisheye cameras.

[0073] Through the above method, based on the common and easily available parameters of roadside scenes, a better solution for obtaining the initial value of external parameters is provided, so that the accuracy is better and it is conducive to subsequent optimization in calibration.

[0074] Different from the method of using vanishing points to calculate initial values ​​in related technologies, this method only uses single information of the vanishing points where straight lines intersect in the image, and uses more common information to determine more degrees of freedom, thereby obtaining the calculation of camera extrinsics.

[0075] In one embodiment of the present application, determining the extrinsic parameters of the camera's coordinate system in the local coordinate system based on the local coordinate system includes: calculating, based on the local coordinate system, a rotation matrix from the local coordinate system to the camera coordinate system and a translation vector from the local coordinate system to the camera coordinate system; determining the extrinsic parameters of the camera's coordinate system in the local coordinate system according to the rotation matrix from the local coordinate system to the camera coordinate system and the translation vector from the local coordinate system to the camera coordinate system.

[0076] In specific implementation, the local coordinate system is established, and the rotation matrix R_c_l from the local coordinate system to the camera coordinate system can be calculated. The translation vector t_c_l from the local coordinate system to the camera coordinate system is equal to the representation of the vector OI in the camera coordinate system. At this point, the external parameter T_c_l of the camera in the local coordinate system is obtained = (R_c_l, t_c_l).

[0077] In one embodiment of the present application, determining the extrinsic parameters of the local coordinate system in the world coordinate system based on the extrinsic parameters of the camera's coordinate system in the local coordinate system includes: calculating the rotation matrix between the world coordinate system and the local coordinate system based on the extrinsic parameters of the camera's coordinate system in the local coordinate system; determining the translation vector between the local coordinate system and the world coordinate system based on the position of the coordinate position of the camera's pole in the world coordinate system corresponding to the lane line corresponding to the high-precision map; determining the extrinsic parameters of the local coordinate system in the world coordinate system based on the rotation matrix between the world coordinate system and the local coordinate system and the translation vector between the local coordinate system and the world coordinate system.

[0078] In specific implementation, the camera's external parameter T_c_l = (R_c_l, t_c_l) in the local coordinate system is used to calculate the rotation matrix R_u_l between the world coordinate system and the local coordinate system, and then the corresponding position of the startpoint on the line CD is used to determine the translation vector t_u_l between the local coordinate system and the world coordinate system. At this point, the local external parameter T_u_l = (R_u_l, t_u_l) in the world coordinate system is obtained.

[0079] It should be noted that the world coordinate system includes the UTM coordinate system.

[0080] In one embodiment of the present application, the determining of the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system includes: determining the extrinsic parameters of the camera in the world coordinate system = the extrinsic parameters of the local coordinate system in the world coordinate system * the extrinsic parameters of the local coordinate system in the camera coordinate system-1 .

[0081] In specific implementation, based on the external parameter T_u_l of the local in the world coordinate system = (R_u_l, t_u_l), the external parameter T_u_c of the camera in the world coordinate system = T_u_l*(T_c_l) can be obtained. -1 .

[0082] In one embodiment of the present application, determining the unit vector passing through O and parallel to the lane line according to the geometric relationship includes: obtaining the unit vector extending outward from point O according to the camera imaging model; calculating the normal vector of the target plane according to the unit vector extending outward from point O; and calculating the unit vector passing through O and parallel to the lane line according to the normal vector of the target plane. It can be understood that the unit vector refers to: Figure 3 The optical center of the camera in the image points to the direction vectors of the real physical points represented by the selected points A, B, C, D, E, and F in the image.

[0083] Firstly, the unit vectors ea, eb, ec, ed, ee, ef along OA, OB, OC, OD, OE, and OF are obtained according to the camera imaging model.

[0084] Secondly, calculate the normal vectors of planes OAB, OCD, and OEF:

[0085] m_oab=ea X eb,m_ocd=ec X ed,m_oef=ee X ef,

[0086] Here, "X" represents the vector cross product.

[0087] Finally, the unit vector n=m_oab×m_ocd passing through O and parallel to the lane line is calculated.

[0088] In one embodiment of the present application, the coordinates of a point on the intersection line of the plane where the lane line is located and its perpendicular plane passing through point O in the camera coordinate system are used to establish a local coordinate system, including: according to the coordinates of the point on the intersection line of the plane where the lane line is located and its perpendicular plane passing through point O in the camera coordinate system, with the first direction as the x-axis, the second direction as the y-axis, and the direction of the first direction vector cross-multiplied by the second direction as the z-axis, establishing the local coordinate system, the first direction and the second direction being obtained according to the geometric relationship.

[0089] When implementing it, Figure 3 As shown, let the lengths of OG, OH, OI be k1, k2, k3. According to |HG|=d1, |IH|=d2, k1, k2, k3 can be obtained, and the coordinates of I, H, G in the camera coordinate system can be known. In general scenes, d1=d2=d can be approximated, thereby simplifying the solution.

[0090] like Figure 3 As shown in the figure, a local coordinate system is established with the IG direction as the x-axis, the ID direction as the y-axis, and the IG x ID direction as the z-axis. The rotation matrix R_c_l from the local coordinate system to the camera coordinate system can be calculated.

[0091] In one embodiment of the present application, a geometric relationship between a camera and a lane line is established based on pre-acquired camera parameters, lane line information, and lane information, including: a geometric relationship between a camera and a lane line is established based on pre-acquired camera intrinsic parameters, information on at least three parallel lane lines, and lane line width, wherein the lane line information includes pixel coordinates on the lane line.

[0092] like Figure 3 As shown in the figure, the camera internal parameters. The camera internal parameters are fixed after leaving the factory. If the factory parameters of the camera are known, the camera internal parameters can be calculated.

[0093] Specifically, there are three parallel lane lines. Figure 1 As shown, each lane line is described by any two points on the lane line center axis in the image. And it is known that Figure 1 The pixel coordinates of A, B, C, D, E, and F in the figure are as follows: Figure 1 The leftmost lane line CD shown, and any point on any lane on the non-CD lane line here.

[0094] The present application also provides a camera extrinsic parameter initial value calculation device 400, such as Figure 4 As shown, a schematic diagram of the structure of a camera extrinsic parameter initial value calculation device in an embodiment of the present application is provided, and the camera extrinsic parameter initial value calculation device 400 at least includes: a preprocessing module 410, a first determination module 420, a calculation module 430, a second determination module 440, a third determination module 450 and a smoothing module 460, wherein:

[0095] In one embodiment of the present application, the preprocessing module 410 is specifically used to establish a geometric relationship between the camera and the lane line according to pre-acquired camera parameters, lane line information and lane information.

[0096] According to the actual scenario of installing cameras on roadside poles, camera parameters, lane line information and lane information are acquired in advance. It can be understood that camera parameters mainly refer to camera internal parameters. Lane line information includes multiple parallel lane lines and corresponding lane lines in high-precision map data. Lane information mainly refers to lane width, which is usually equal.

[0097] It should be noted that the camera installed on the roadside pole can be a box camera or a fisheye camera.

[0098] According to the pre-obtained camera parameters, lane line information and lane information, the following can be established: Figure 3 The camera and lane geometry shown.

[0099] In one embodiment of the present application, the preprocessing module 420 is specifically used to: determine a unit vector passing through O and parallel to the lane line according to the geometric relationship, where O is the center of the camera model after dedistortion.

[0100] Please continue to refer to Figure 3 ,exist Figure 3 AB, CD, EF are the right, left, and center lane lines respectively. The pixel coordinate information of each point is known. The same symbol is used to represent the same physical point in the image and three-dimensional space. O is the center of the camera pinhole model after dedistortion. I, H, G are the points on the intersection of the plane where the lane line is located and the perpendicular plane passing through point O.

[0101] According to the geometric relationship, the normal vector of the target plane is determined. A unit vector passing through O and parallel to the lane line can also be determined.

[0102] In one embodiment of the present application, the preprocessing module 430 is specifically used to calculate the coordinates of the point on the intersection line of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system based on the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information. The coordinates of the point on the intersection line of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system are used to establish a local coordinate system.

[0103] According to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, the coordinates of the point on the intersection of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system are further calculated, as follows: Figure 3 As shown, the coordinates of I, H, and G in the cam coordinate system can be obtained accordingly.

[0104] In one embodiment of the present application, the preprocessing module 440 is specifically used to determine the external parameters of the camera's local coordinate system based on the local coordinate system.

[0105] The local coordinate system can be obtained based on the coordinates of I, H, and G in the cam coordinate system, and then the external parameters of the camera's local coordinate system can be determined based on the local coordinate system.

[0106] In one embodiment of the present application, the preprocessing module 450 is specifically used to determine the external parameters of the local coordinate system in the world coordinate system based on the high-precision map data of the target area.

[0107] The normal vector of the target plane can be obtained through the high-precision map data of the target area.

[0108] According to the normal vector of the target plane, lane line information, and starting point information provided by the high-precision map data of the target area, the external parameter parameters of the local coordinate system in the world coordinate system are determined.

[0109] In one embodiment of the present application, the preprocessing module 460 is specifically used to determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

[0110] Finally, according to the extrinsic parameters (rotation matrix and translation vector) of the local coordinate system in the world coordinate system, the extrinsic parameters of the camera in the world coordinate system are determined.

[0111] In one embodiment of the present application, the second determining module 430 is further configured to:

[0112] Based on the local coordinate system, a rotation matrix from the local coordinate system to the camera coordinate system and a translation vector from the local coordinate system to the camera coordinate system are calculated;

[0113] Determine the extrinsic parameters of the camera in the local coordinate system according to the rotation matrix from the local coordinate system to the camera coordinate system and the translation vector from the local coordinate system to the camera coordinate system.

[0114] In one embodiment of the present application, the third determination module 440 is further configured to:

[0115] Calculate the rotation matrix between the world coordinate system and the local coordinate system according to the external parameters of the camera in the local coordinate system;

[0116] Determine the translation vector between the local coordinate system and the world coordinate system based on the coordinate position of the pole where the camera is located in the world coordinate system and the corresponding position on the lane line corresponding to the high-precision map;

[0117] According to the rotation matrix between the world coordinate system and the local coordinate system and the translation vector between the local coordinate system and the world coordinate system, the extrinsic parameters of the local coordinate system in the world coordinate system are determined.

[0118] In one embodiment of the present application, the fourth determination module 460 is further configured to:

[0119] Determine the external parameter parameters of the camera in the world coordinate system = the external parameter parameters of the local coordinate system in the world coordinate system * the external parameter parameters of the local coordinate system in the camera coordinate system -1.

[0120] In one embodiment of the present application, the first determining module is further configured to:

[0121] According to the camera imaging model, obtain the unit vector extending outward from point O;

[0122] Calculate the normal vector of the target plane according to the unit vector extending outward from point O;

[0123] According to the normal vector of the target plane, a unit vector passing through O and parallel to the lane line is calculated.

[0124] In one embodiment of the present application, the calculation module 420 is also used to

[0125] According to the coordinates of the points on the intersection of the plane where the lane line is located and the perpendicular plane passing through point O in the camera coordinate system, the local coordinate system is established with the first direction as the x-axis, the second direction as the y-axis, and the direction of the cross product of the first direction vector and the second direction as the z-axis. The first direction and the second direction are obtained according to the geometric relationship.

[0126] In one embodiment of the present application, the preprocessing module 410 is also used to

[0127] A geometric relationship between the camera and the lane line is established based on pre-acquired camera intrinsic parameters, information of at least three parallel lane lines, and lane line widths, wherein the lane line information includes pixel coordinates located on the lane line.

[0128] It can be understood that the above-mentioned camera extrinsic parameter initial value calculation device can implement the various steps of the camera extrinsic parameter initial value calculation method provided in the aforementioned embodiment, and the relevant explanations about the camera extrinsic parameter initial value calculation method are applicable to the camera extrinsic parameter initial value calculation device, which will not be repeated here.

[0129] Figure 5 This is a schematic diagram of the structure of an electronic device according to an embodiment of the present application. Figure 5 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and a memory. The memory may include a memory, such as a high-speed random access memory (RAM), and may also include a non-volatile memory (non-volatile memory), such as at least one disk storage. Of course, the electronic device may also include hardware required for other services.

[0130] The processor, network interface and memory can be interconnected through an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 5 Only one bidirectional arrow is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0131] The memory is used to store the program. Specifically, the program may include a program code, and the program code includes a computer operation instruction. The memory may include a memory and a non-volatile memory, and provides instructions and data to the processor.

[0132] The processor reads the corresponding computer program from the non-volatile memory into the memory and then runs it, forming a camera external parameter initial value calculation device at the logical level. The processor executes the program stored in the memory and is specifically used to perform the following operations:

[0133] According to the pre-acquired camera parameters, lane line information and lane information, a geometric relationship between the camera and the lane line is established;

[0134] According to the geometric relationship, determine a unit vector passing through O and parallel to the lane line, where O is the center of the camera model after dedistortion;

[0135] According to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are calculated, and the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system;

[0136] Based on the local coordinate system, determine the external parameters of the camera in the local coordinate system;

[0137] Determine the external parameters of the local coordinate system in the world coordinate system based on the high-precision map data of the target area;

[0138] Determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

[0139] The above application Figure 2The method performed by the camera extrinsic initial value calculation device disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by an integrated logic circuit of hardware in the processor or instructions in software form. The above processor may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc. The steps of the method disclosed in conjunction with the embodiments of the present application can be directly embodied as being executed by a hardware decoding processor, or executed by a combination of hardware and software modules in a decoding processor. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0140] The electronic device may also perform Figure 2 The method executed by the camera extrinsic parameter initial value calculation device in the embodiment of the present invention is realized Figure 2 The functions of the illustrated embodiment will not be described in detail in the embodiments of the present application.

[0141] The present application also provides a computer-readable storage medium, which stores one or more programs, wherein the one or more programs include instructions, which, when executed by an electronic device including multiple application programs, enable the electronic device to execute Figure 2 The method executed by the camera extrinsic parameter initial value calculation device in the embodiment shown is specifically used to execute:

[0142] According to the pre-acquired camera parameters, lane line information and lane information, a geometric relationship between the camera and the lane line is established;

[0143] According to the geometric relationship, determine a unit vector passing through O and parallel to the lane line, where O is the center of the camera model after dedistortion;

[0144] According to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are calculated, and the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system;

[0145] Based on the local coordinate system, determine the external parameters of the camera in the local coordinate system;

[0146] Determine the external parameters of the local coordinate system in the world coordinate system based on the high-precision map data of the target area;

[0147] Determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

[0148] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0149] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0150] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0151] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0152] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0153] The memory may include non-permanent storage in a computer-readable medium, random access memory (RAM) and / or non-volatile memory in the form of read-only memory (ROM) or flash RAM. The memory is an example of a computer-readable medium.

[0154] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0155] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0156] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that contain computer-usable program code.

[0157] The above is only an embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included in the scope of the claims of the present application.

Claims

1. A camera extrinsic parameter calculation method, wherein: The method comprises: According to the pre-acquired camera parameters, lane line information and lane information, a geometric relationship between the camera and the lane line is established; According to the geometric relationship, determine a unit vector passing through O and parallel to the lane line, where O is the center of the camera model after dedistortion; According to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are calculated, and the coordinates of the points on the intersection line of the plane where the lane line is located and the vertical plane passing through point O in the camera coordinate system are used to establish the local coordinate system; Based on the local coordinate system, determine the external parameters of the camera in the local coordinate system; Determine the external parameters of the local coordinate system in the world coordinate system based on the high-precision map data of the target area; Determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

2. The method of claim 1, wherein: The determining, based on the local coordinate system, external parameters of the camera in the local coordinate system includes: Based on the local coordinate system, a rotation matrix from the local coordinate system to the camera coordinate system and a translation vector from the local coordinate system to the camera coordinate system are calculated; Determine the extrinsic parameters of the camera in the local coordinate system according to the rotation matrix from the local coordinate system to the camera coordinate system and the translation vector from the local coordinate system to the camera coordinate system.

3. The method of claim 2, wherein: Determining the external parameters of the local coordinate system in the world coordinate system according to the high-precision map data of the target area includes: Calculate the rotation matrix and translation vector between the world coordinate system and the local coordinate system based on the plane normal vector and lane line information provided in the high-precision map data of the target area; Determine the translation vector between the local coordinate system and the world coordinate system based on the coordinate position of the pole where the camera is located in the world coordinate system and the corresponding position on the lane line corresponding to the high-precision map; According to the rotation matrix between the world coordinate system and the local coordinate system and the translation vector between the local coordinate system and the world coordinate system, the extrinsic parameters of the local coordinate system in the world coordinate system are determined.

4. The method of claim 1, wherein: The determining the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system includes: Determine the external parameter parameters of the camera in the world coordinate system = the external parameter parameters of the local coordinate system in the world coordinate system * the external parameter parameters of the local coordinate system in the camera coordinate system -1 .

5. The method of claim 1, wherein: Determining, according to the geometric relationship, a unit vector passing through O and parallel to the lane line includes: According to the camera imaging model, obtain the unit vector extending outward from point O; Calculate the normal vector of the target plane according to the unit vector extending outward from point O; According to the normal vector of the target plane, a unit vector passing through O and parallel to the lane line is calculated.

6. The method of claim 5, wherein: The coordinates of the points on the intersection of the plane where the lane line is located and the perpendicular plane passing through point O in the camera coordinate system are used to establish the local coordinate system, including: According to the coordinates of the points on the intersection of the plane where the lane line is located and the perpendicular plane passing through point O in the camera coordinate system, the local coordinate system is established with the first direction as the x-axis, the second direction as the y-axis, and the direction of the cross product of the first direction vector and the second direction as the z-axis. The first direction and the second direction are obtained according to the geometric relationship.

7. The method of claim 1, wherein: According to the pre-acquired camera parameters, lane line information and lane information, the geometric relationship between the camera and the lane line is established, including: A geometric relationship between the camera and the lane line is established based on pre-acquired camera intrinsic parameters, information of at least three parallel lane lines, and lane line widths, wherein the lane line information includes pixel coordinates located on the lane line.

8. A camera extrinsic parameter initial value calculation device, wherein: The device comprises: A preprocessing module, used to establish a geometric relationship between the camera and the lane line according to the pre-acquired camera parameters, lane line information and lane information; A first determination module is used to determine a unit vector passing through O and parallel to the lane line according to the geometric relationship, where O is the center of the camera model after dedistortion; A calculation module, for calculating the coordinates of a point on the intersection line of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system according to the normal vector of the target plane, the unit vector passing through point O and parallel to the lane line, and the lane information, wherein the coordinates of the point on the intersection line of the plane where the lane line is located and its vertical plane passing through point O in the camera coordinate system are used to establish a local coordinate system; A second determination module is used to determine the external parameters of the camera in the local coordinate system based on the local coordinate system; The third determination module is used to determine the external parameters of the local coordinate system in the world coordinate system according to the high-precision map data of the target area; The fourth determination module is used to determine the extrinsic parameters of the camera in the world coordinate system according to the extrinsic parameters of the local coordinate system in the world coordinate system and the extrinsic parameters of the camera in the local coordinate system.

9. An electronic device, comprising: processor; as well as A memory arranged to store computer executable instructions, which when executed cause the processor to perform the method of any one of claims 1 to 7.

10. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of application programs, causes the electronic device to execute any one of the methods of claims 1 to 7.

Images