Camera calibration method and device, electronic device, and storage medium
By using a dynamic Euclidean distance optimization function to iteratively calculate the extrinsic parameter matrix in roadside camera calibration, the problem of low 2D-3D mapping accuracy of roadside cameras was solved, and higher positioning accuracy was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- ZHIDAO NETWORK TECH (BEIJING) CO LTD
- Filing Date
- 2022-12-08
- Publication Date
- 2026-05-26
Smart Images
Figure CN115761010B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of camera calibration technology, and in particular to a camera calibration method, apparatus, electronic device, and storage medium. Background Technology
[0002] Roadside cameras installed on smart roads or at intersections need to be calibrated before they can be used for target identification and localization. During calibration, road markings or calibration boards are usually used, and 3D-2D calibration algorithms are employed to calculate the extrinsic parameters of the roadside cameras.
[0003] In related technologies, the 3D-2D calibration algorithm mainly uses PNP (Perspective-n-Point) to calculate the extrinsic parameter matrix. However, when using PNP calculation, since the roadside camera is calibrated as a 2D-3D mapping, the mapping accuracy from 2D to 3D is not high, which further affects the subsequent positioning accuracy. Summary of the Invention
[0004] This application provides a camera calibration method, apparatus, electronic device, and storage medium to optimize the extrinsic parameter matrix of roadside cameras and improve the 3D positioning accuracy of roadside cameras from 2D to 3D.
[0005] The embodiments of this application adopt the following technical solutions:
[0006] In a first aspect, embodiments of this application provide a camera calibration method, wherein the method includes:
[0007] Collect multiple point pairs of corresponding 2D points in images captured by a camera and 3D points in a real scene to obtain the first calibration parameter. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points.
[0008] The second calibration parameters are obtained by iterating the first calibration parameters according to the preset optimization function;
[0009] Based on the second calibration parameters, a mapping table between the 2D points and the 3D points is determined.
[0010] In some embodiments, the acquisition of multiple corresponding point pairs of 2D points and 3D points to obtain the first calibration parameters includes:
[0011] Based on the PNP algorithm, the initial extrinsic parameter matrix [R|t] and the initial height d of the camera from the ground are calculated.
[0012] Based on the coordinates of the 2D point, the coordinates of the 3D point in the world coordinate system are calculated according to the initialization extrinsic parameter matrix [R|t].
[0013] The initialization extrinsic parameter matrix [R|t], the initial height d of the camera from the ground, and the coordinate position of the 3D point in the world coordinate system are used as the first calibration parameters.
[0014] In some embodiments, the method further includes:
[0015] By filtering the coordinates of the 3D points in the world coordinate system and the initial height d of the camera from the ground, the initial normal vector n of the road plane in the ENU coordinate system is determined.
[0016] In some embodiments, the step of iterating the first calibration parameters according to a preset optimization function to obtain the second calibration parameters includes:
[0017] The rotation matrix R from the camera to the ENU and the offset vector t from the origin of the camera coordinate system to the origin of the ENU coordinate system are iterated in the initialization extrinsic parameter matrix [R|t] of the first calibration parameters according to the preset optimization function.
[0018] Iterate the normal vector n of the road plane in the ENU coordinate system in the first calibration parameters according to the preset optimization function;
[0019] The optimized and iterated R, t, and n are used as the second calibration parameters.
[0020] In some embodiments, the preset optimization function includes:
[0021] min(1 / m∑(di / max_d)|Pc-Pt|)
[0022] m is the number of corresponding point pairs between the 2D point and the 3D point, di is the Euclidean distance between the 3D point and the camera origin, max_d is the Euclidean distance between the point farthest from the camera among the multiple corresponding point pairs between the 2D point and the 3D point, Pc = [R|t]*Pi, where Pc is the 3D point, Pi is the 2D point, and Pt is the point in the ENU coordinate system.
[0023] In some embodiments, the preset optimization function further includes: using the optimized and iterated R, t, and n as the second calibration parameters and minimizing the value of the preset optimization function.
[0024] In some embodiments, the acquisition of multiple corresponding point pairs of 2D points and 3D points includes: acquiring at least three multiple corresponding point pairs of 2D points and 3D points.
[0025] In some embodiments, the pose parameters of the roadside camera are obtained according to the mapping table between 2D points and 3D points;
[0026] Based on the pose parameters of the roadside camera, the latitude, longitude, and height parameters of the target object on the road surface from the perspective of the roadside camera are determined to provide roadside positioning information.
[0027] Secondly, embodiments of this application also provide a camera calibration device, wherein the device includes:
[0028] The acquisition module is used to acquire multiple point pairs of corresponding 2D points in images captured by a camera and 3D points in a real scene to obtain a first calibration parameter. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points.
[0029] The iteration module is used to iterate the first calibration parameters according to a preset optimization function to obtain the second calibration parameters;
[0030] The determination module is used to determine the mapping relationship table between the 2D points and the 3D points based on the second calibration parameters.
[0031] Thirdly, embodiments of this application also provide an electronic device, including: a processor; and a memory arranged to store computer-executable instructions, which, when executed, cause the processor to perform the above-described method.
[0032] Fourthly, embodiments of this application also provide a computer-readable storage medium that stores one or more programs, which, when executed by an electronic device including multiple applications, cause the electronic device to perform the above-described method.
[0033] The above-mentioned at least one technical solution adopted in the embodiments of this application can achieve the following beneficial effects: by obtaining the first calibration parameter and iterating through the constructed optimization function, the optimized result is obtained as the second calibration parameter. The mapping relationship table between 2D points and 3D points determined according to the second calibration parameter is more accurate, thereby improving the 2D to 3D mapping accuracy on the road plane where the roadside camera is located, thereby improving the positioning accuracy (target latitude and longitude information, target height information) of the roadside camera for target objects (vehicles, pedestrians) in the road. Attached Figure Description
[0034] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:
[0035] Figure 1 This is a schematic diagram of the camera calibration method in the embodiments of this application;
[0036] Figure 2This is a schematic diagram of the camera calibration device structure in an embodiment of this application;
[0037] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Detailed Implementation
[0038] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0039] The technical terms used in this application's embodiments are as follows:
[0040] UTM, or Universal Transverse Mercator Grid System, is a type of Cartesian coordinate system.
[0041] PnP, an n-point perspective transformation algorithm.
[0042] ENU, the Northeast Celestial Rectangular Coordinate System.
[0043] During their research, the inventors discovered that the commonly used method for calibration from the camera coordinate system to the UTM coordinate system is the 3D-2D PnP algorithm. The optimization function of PnP directly ensures that the reprojection error of 3D points projected onto the image is minimized. However, the optimization function used here is not directly and strongly correlated with the positioning accuracy, thus affecting the mapping accuracy of the final 2D-3D mapping relationship.
[0044] Furthermore, the 3D-2D PnP calibration method ensures 3D-to-2D projection accuracy by minimizing the reprojection error of pixels in the image. However, if directly applied to roadside cameras, the 2D-to-3D mapping accuracy is not high.
[0045] To address the aforementioned shortcomings, the camera calibration method employed in the embodiments of this application optimizes and iteratively calculates the extrinsic parameter matrix based on a dynamic Euclidean distance optimization function, thereby improving the 3D positioning accuracy of roadside cameras in detecting targets from 2D to 3D in real-world scenes.
[0046] The technical solutions provided by the various embodiments of this application are described in detail below with reference to the accompanying drawings.
[0047] This application provides a camera calibration method, such as... Figure 1 The diagram shows a schematic flowchart of a camera calibration method in an embodiment of this application. The method includes at least the following steps S110 to S140:
[0048] Step S110: Collect multiple point pairs containing 2D points in images captured by the camera and corresponding 3D points in the real scene to obtain the first calibration parameters. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points.
[0049] The example of a roadside camera is used for illustration and is not intended to limit the specific type.
[0050] Roadside cameras refer to a large number of perception sensors deployed on smart roads, serving as the main devices among these sensors. After identifying a target, a roadside camera needs to obtain the target's world coordinates (latitude and longitude).
[0051] However, before roadside camera identification and positioning, the coordinate system of the roadside camera needs to be converted to the UTM coordinate system, which is the camera-UTM calibration process. The calibration process completes the function of converting the camera pixel position to latitude and longitude, and optimizes the accuracy.
[0052] In some embodiments, the preset longitudinal (parallel to the road) monitoring range of the roadside camera is 80m-100m. The calibration results of the roadside camera must ensure that the positioning accuracy of the target on the road does not deviate from the lane, and that the vehicles in front and behind do not overlap or pass through the lane. The positioning accuracy is ensured by the roadside camera calibration task.
[0053] The calibration process of a roadside camera-UTM involves converting 2D points into 3D points. First, it requires acquiring multiple corresponding point pairs between 2D points in the images captured by the roadside camera and 3D points in the actual roadside scene. In other words, it involves matching the coordinate positions of points in the world coordinate system with their corresponding coordinate positions in the pixel coordinate system.
[0054] In some embodiments, when using the PNP algorithm to solve the problem, the step of collecting multiple corresponding point pairs of 2D points and 3D points includes: collecting at least three multiple corresponding point pairs of 2D points and 3D points. Preferably, multiple corresponding point pairs of four 2D points and 3D points can be collected.
[0055] The above-mentioned multiple corresponding point pairs are used as the first calibration parameters.
[0056] The first calibration parameters also need to include the extrinsic parameter matrix obtained after initialization using the PNP algorithm and the height information of the roadside camera from the current road surface.
[0057] Step S120: Iterate the first calibration parameters according to the preset optimization function to obtain the second calibration parameters.
[0058] An optimization function is constructed to further ensure the 2D-3D projection accuracy of the roadside camera. The optimization function iteratively calculates the first calibration parameters obtained in the above steps to obtain the second calibration parameters. In other words, to obtain the optimal solution, the optimization function is needed to iterate over the first calibration parameters.
[0059] In some embodiments, in order to improve the accuracy of 2D-3D calibration, the height of the roadside camera from the actual road surface is determined based on Euclidean distance when constructing the optimization function.
[0060] In some embodiments, in order to improve the 2D-3D calibration accuracy, an optimization function with dynamic weights is considered when constructing the optimization function.
[0061] Step S130: Determine the mapping relationship table between the 2D points and the 3D points according to the second calibration parameters.
[0062] The second calibration parameter is the parameter after iteration, thereby determining the mapping relationship table between the 2D point and the 3D point. Based on the mapping relationship table between the 2D point and the 3D point, the mapping relationship between the 2D point and the 3D point can be obtained, thereby realizing the calibration of the roadside camera-UTM.
[0063] The camera calibration method in this application embodiment calculates the initialized camera extrinsic parameter matrix, then constructs an optimization function to iterate the camera extrinsic parameter model, obtains the optimized result, and then determines the mapping relationship between the 2D points and 3D points of the roadside camera.
[0064] Unlike related technologies that use 3D-2D PNP algorithms, the PNP algorithm's optimization function directly ensures that the reprojection error of 3D points projected onto the image is minimized. However, the PNP algorithm's optimization function is not directly and strongly correlated with the target's positioning accuracy, which can easily lead to a decrease in accuracy. In this application, the 3D positioning accuracy of the roadside camera's 2D-3D model is improved by optimizing and iteratively calculating the camera's extrinsic parameter matrix and the camera's height above the ground.
[0065] In one embodiment of this application, the acquisition of multiple corresponding point pairs of 2D and 3D points to obtain the first calibration parameter includes: calculating the initial extrinsic parameter matrix [R|t] and the initial height d of the camera from the ground according to the PNP algorithm; calculating the coordinate position of the 3D point in the world coordinate system based on the coordinate position of the 2D point and the initial extrinsic parameter matrix [R|t]; and using the initial extrinsic parameter matrix [R|t], the initial height d of the camera from the ground, and the coordinate position of the 3D point in the world coordinate system as the first calibration parameter.
[0066] After acquiring multiple corresponding point pairs of more than four 2D and 3D points, the initial camera extrinsic parameter matrix and the camera's height above the current road surface are calculated using the PNP algorithm. The initial camera extrinsic parameter matrix is directly calculated using the PNP algorithm based on multiple corresponding point pairs of 2D and 3D points, and this matrix is used as the initial camera extrinsic parameter matrix.
[0067] Simultaneously, the initial height d of the camera above the ground is calculated. Then, based on the initialized camera extrinsic parameter matrix and the coordinate position information of the 2D point, the coordinate position of the 3D point in the world coordinate system is calculated (estimated).
[0068] Finally, the initial extrinsic parameter matrix [R|t], the initial height d of the camera from the ground, and the coordinate position of the 3D point in the world coordinate system are used as the first calibration parameters.
[0069] It can be understood that in the initialization of the extrinsic parameter matrix [R|t], R represents the rotation matrix from the camera to the ENU coordinate system, and t represents the offset vector from the origin of the camera coordinate system to the origin of the ENU coordinate system.
[0070] It's important to note that when calculating the distance from the camera to the road surface, the road surface is typically assumed to be horizontal, i.e., an absolute plane. The unit normal vector of the horizontal plane xoy is {0,0,1}. In the ENU (East-North-Sky) Cartesian coordinate system, EN is the absolute plane, and U points east. However, in real-world scenes, road surfaces are not necessarily horizontal but rather inclined relative to the horizontal plane. Therefore, it's necessary to calculate the angle between the actual road surface and the absolute plane to improve the 2D-to-3D mapping accuracy. Thus, the solution process can iterate through the normal vector n of the road plane in the ENU using an optimization function.
[0071] It can be understood that the normal vector n of the initialized road plane in ENU is the initial height d of the camera from the ground.
[0072] Through the above steps, the initial extrinsic parameter matrix [R|t], the initial height d of the camera from the ground, and the coordinate position of the 3D point in the world coordinate system are used as the first calibration parameters.
[0073] Solve for the coordinates of a 3D point in the world coordinate system, i.e., Pc = [R|t] * Pi, where Pi is a pixel image point selected based on a 2D point, and [R|t] is the initialization extrinsic parameter matrix.
[0074] In one embodiment of this application, the method further includes: filtering the coordinate positions of the 3D points in the world coordinate system and the initial height d of the camera from the ground, and determining the initial normal vector n of the road plane in the ENU coordinate system.
[0075] To find the optimal normal vector n for the road in the ENU coordinate system, the coordinates of the 3D points in the world coordinate system and the initial height d of the camera above the ground are selected. This process is also an important factor in improving the camera's long-range accuracy. Long-range accuracy refers to the accuracy of localization and perception of distant targets.
[0076] In one embodiment of this application, the step of iterating the first calibration parameters according to a preset optimization function to obtain the second calibration parameters includes: iterating the rotation matrix R from the camera to the ENU and the offset vector t from the origin of the camera coordinate system to the origin of the ENU coordinate system in the initialization extrinsic parameter matrix [R|t] of the first calibration parameters according to the preset optimization function, and iterating the normal vector n of the road plane in the ENU coordinate system in the first calibration parameters according to the preset optimization function; and using the optimized R, t, and n as the second calibration parameters.
[0077] After optimizing the rotation matrix R from the camera to the ENU and the offset vector t from the origin of the camera coordinate system to the origin of the ENU coordinate system in the initial extrinsic parameter matrix [R|t] using a preset optimization function, and iterating the normal vector n of the road plane in the ENU coordinate system in the first calibration parameter according to the preset optimization function, the optimized R, t, and n are obtained as the second calibration parameters.
[0078] If we iterate through the three parameters R, t, and n based on the optimization function to minimize the optimization function, we can obtain the optimal solution, which can then be used as the second calibration parameter. Based on the final optimized R, t, and n, we obtain a mapping table from 2D to 3D.
[0079] In one embodiment of this application, the preset optimization function includes:
[0080] min(1 / m∑(d i / max_d)|Pc-Pt|)
[0081] The m represents the number of corresponding point pairs between the 2D point and the 3D point, with m being at least 3, and preferably more than 4.
[0082] The d i Let d be the Euclidean distance between the 3D point and the camera origin, and max_d be the Euclidean distance between the point farthest from the camera among multiple corresponding point pairs of the 2D and 3D points. i / max_d is used as a dynamic weight, and dynamic Euclidean distance is used as the weight information for point pairs to ensure that the weight of distant points of the roadside camera is large and the accuracy is high. At the same time, the pinhole imaging itself ensures the accuracy of nearby points.
[0083] The |Pc-Pt| is used as the absolute difference, where Pc = [R|t]*Pi, Pc is a 3D point, Pi is a 2D point, and Pt is a point in the ENU coordinate system. The optimized R, t, and n are used as the second calibration parameters to minimize the value of the preset optimization function. By directly calculating the extrinsic parameter matrix [R|t] and camera height di using Euclidean distance minimization, the preset optimization function is strongly correlated with accuracy. Unlike related technologies, the optimization function of PNP is not directly strongly correlated with positioning accuracy, thus failing to improve positioning accuracy.
[0084] In one embodiment of this application, the method further includes: obtaining the pose parameters of the roadside camera according to the mapping relationship table between 2D points and 3D points; and determining the latitude, longitude, and height parameters of the target object on the road surface from the perspective of the roadside camera according to the pose parameters of the roadside camera, so as to provide roadside positioning information.
[0085] Based on the mapping table between 2D points and 3D points, the pose parameters of the roadside camera are obtained. According to the pose parameters of the roadside camera, the latitude, longitude and height parameters of the target object on the road surface from the perspective of the roadside camera are determined. Based on the optimized mapping results, the roadside camera can achieve accurate perception.
[0086] In some embodiments, a first calibration parameter is obtained by acquiring multiple corresponding point pairs between 2D points in images captured by roadside cameras and 3D points in a real roadside scene. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points. The first calibration parameter is iterated according to a preset optimization function to obtain a second calibration parameter. The mapping relationship table between the 2D points and the 3D points is determined according to the second calibration parameter.
[0087] The acquisition of multiple corresponding point pairs of 2D and 3D points to obtain the first calibration parameters includes: calculating the initial extrinsic parameter matrix [R|t] and the initial height d of the camera from the ground according to the PNP algorithm; calculating the coordinate position of the 3D point in the world coordinate system based on the coordinate position of the 2D point and the initial extrinsic parameter matrix [R|t]; and using the initial extrinsic parameter matrix [R|t], the initial height d of the camera from the ground, and the coordinate position of the 3D point in the world coordinate system as the first calibration parameters.
[0088] The method further includes: filtering the coordinate positions of the 3D points in the world coordinate system and the initial height d of the camera from the ground, and determining the initial normal vector n of the road plane in the ENU coordinate system.
[0089] The step of iterating the first calibration parameters according to a preset optimization function to obtain the second calibration parameters includes: iterating the rotation matrix R from the camera to the ENU and the offset vector t from the origin of the camera coordinate system to the origin of the ENU coordinate system in the initialization extrinsic parameter matrix [R|t] of the first calibration parameters according to the preset optimization function, and iterating the normal vector n of the road plane in the ENU coordinate system in the first calibration parameters according to the preset optimization function; and using the optimized R, t, and n as the second calibration parameters.
[0090] The preset optimization function includes: min(1 / m∑(d i / max_d)|Pc-Pt|)
[0091] m is the number of corresponding point pairs between the 2D point and the 3D point, and d i The Euclidean distance between the 3D point and the camera origin is given by _d, where max_d is the Euclidean distance between the point farthest from the camera among multiple corresponding point pairs of the 2D and 3D points. Pc = [R|t]*Pi, where Pc is the 3D point, Pi is the 2D point, and Pt is a point in the ENU coordinate system. The optimized R, t, and n are used as the second calibration parameters to minimize the value of the preset optimization function.
[0092] By minimizing the Euclidean distance, the extrinsic parameter matrices R, t, and camera height d can be directly obtained, making the optimization function strongly correlated with the accuracy.
[0093] Using dynamic Euclidean distance as the weight information for point pairs ensures that distant points have a high weight and high accuracy, while pinhole imaging itself ensures accuracy at close range.
[0094] Finding the optimal path normal vector n in ENU is a major factor in improving accuracy at long distances.
[0095] This application embodiment also provides a camera calibration device 200, such as Figure 2 The diagram shows a schematic representation of a camera calibration device in an embodiment of this application. The camera calibration device 200 includes at least: an acquisition module 210, an iteration module 220, and a determination module 230, wherein:
[0096] In one embodiment of this application, the acquisition module 210 is specifically used to: acquire multiple point pairs containing corresponding 2D points in images captured by a camera and 3D points in a real scene to obtain a first calibration parameter, wherein the 3D points include coordinate position points in the world coordinate system and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points.
[0097] The example of a roadside camera is used for illustration and is not intended to limit the specific type.
[0098] Roadside cameras refer to a large number of perception sensors deployed on smart roads, serving as the main devices among these sensors. After identifying a target, a roadside camera needs to obtain the target's world coordinates (latitude and longitude).
[0099] However, before roadside camera identification and positioning, the coordinate system of the roadside camera needs to be converted to the UTM coordinate system, which is the camera-UTM calibration process. The calibration process completes the function of converting the camera pixel position to latitude and longitude, and optimizes the accuracy.
[0100] In some embodiments, the preset longitudinal (parallel to the road) monitoring range of the roadside camera is 80m-100m. The calibration results of the roadside camera must ensure that the positioning accuracy of the target on the road does not deviate from the lane, and that the vehicles in front and behind do not overlap or pass through the lane. The positioning accuracy is ensured by the roadside camera calibration task.
[0101] The calibration process from roadside camera to UTM involves converting 2D points into 3D points. First, it requires collecting multiple corresponding point pairs between 2D points in the roadside camera's image and 3D points in the actual roadside scene. In other words, it involves matching the coordinate positions of points in the world coordinate system with their corresponding coordinate positions in the pixel coordinate system.
[0102] In some embodiments, when using the PNP algorithm to solve the problem, the step of collecting multiple corresponding point pairs of 2D points and 3D points includes: collecting at least three multiple corresponding point pairs of 2D points and 3D points. Preferably, multiple corresponding point pairs of four 2D points and 3D points can be collected.
[0103] The above-mentioned multiple corresponding point pairs are used as the first calibration parameters.
[0104] The first calibration parameters also need to include the extrinsic parameter matrix obtained after initialization using the PNP algorithm and the height information of the roadside camera from the current road surface.
[0105] In one embodiment of this application, the iteration module 220 is specifically used to: iterate the first calibration parameters according to a preset optimization function to obtain the second calibration parameters.
[0106] An optimization function is constructed to further ensure the 2D-3D projection accuracy of the roadside camera. The optimization function iteratively calculates the first calibration parameters obtained in the above steps to obtain the second calibration parameters. In other words, to obtain the optimal solution, the optimization function is needed to iterate over the first calibration parameters.
[0107] In some embodiments, in order to improve the accuracy of 2D-3D calibration, the height of the roadside camera from the actual road surface is determined based on Euclidean distance when constructing the optimization function.
[0108] In some embodiments, in order to improve the 2D-3D calibration accuracy, an optimization function with dynamic weights is considered when constructing the optimization function.
[0109] In one embodiment of this application, the determining module 230 is specifically used to: determine the mapping relationship table between the 2D points and the 3D points according to the second calibration parameters.
[0110] The second calibration parameter is the parameter after iteration, thereby determining the mapping relationship table between the 2D point and the 3D point. Based on the mapping relationship table between the 2D point and the 3D point, the mapping relationship between the 2D point and the 3D point can be obtained, thereby realizing the calibration of the roadside camera-UTM.
[0111] It is understood that the camera calibration device described above can implement each step of the camera calibration method provided in the foregoing embodiments. The relevant explanations of the camera calibration method are applicable to the camera calibration device and will not be repeated here.
[0112] Figure 3 This is a schematic diagram of the structure of an electronic device according to an embodiment of this application. Please refer to it. Figure 3 At the hardware level, the electronic device includes a processor, and optionally also includes an internal bus, a network interface, and memory. The memory may include main memory, such as high-speed random-access memory (RAM), or non-volatile memory, such as at least one disk drive. Of course, the electronic device may also include other hardware required for other business operations.
[0113] The processor, network interface, and memory can be interconnected via an internal bus, which can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 3 The symbol is represented by a single double-headed arrow, but this does not mean that there is only one bus or one type of bus.
[0114] Memory is used to store programs. Specifically, programs may include program code, which includes computer operation instructions. Memory may include main memory and non-volatile memory, and provides instructions and data to the processor.
[0115] The processor reads the corresponding computer program from non-volatile memory into main memory and then executes it, forming a camera calibration device at the logical level. The processor executes the program stored in memory and specifically performs the following operations:
[0116] Collect multiple point pairs of corresponding 2D points in images captured by a camera and 3D points in a real scene to obtain the first calibration parameter. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points.
[0117] The second calibration parameters are obtained by iterating the first calibration parameters according to the preset optimization function;
[0118] Based on the second calibration parameters, a mapping table between the 2D points and the 3D points is determined.
[0119] The above is as stated in this application. Figure 1 The camera calibration device method disclosed in the illustrated embodiment can be applied to a processor or implemented by a processor. The processor may be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor can be a general-purpose processor, including a Central Processing Unit (CPU), a Network Processor (NP), etc.; it can also be a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), a Field-Programmable Gate Array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software module can reside in a mature storage medium in the field, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.
[0120] The electronic device can also perform Figure 1 The method for executing the camera calibration device, and the realization of the camera calibration device in Figure 1 The functions of the embodiments shown are not described in detail here.
[0121] This application also proposes a computer-readable storage medium that stores one or more programs, the programs including instructions that, when executed by an electronic device including multiple applications, enable the electronic device to perform... Figure 1 The method executed by the camera calibration device in the illustrated embodiment is specifically used to perform:
[0122] Collect multiple point pairs of corresponding 2D points in images captured by a camera and 3D points in a real scene to obtain the first calibration parameter. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points.
[0123] The second calibration parameters are obtained by iterating the first calibration parameters according to the preset optimization function;
[0124] Based on the second calibration parameters, a mapping table between the 2D points and the 3D points is determined.
[0125] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0126] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.
[0127] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.
[0128] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.
[0129] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.
[0130] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.
[0131] Computer-readable media includes both permanent and non-permanent, removable and non-removable media that can store information using any method or technology. Information can be computer-readable instructions, data structures, modules of programs, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.
[0132] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0133] Those skilled in the art will understand that embodiments of this application can be provided as methods, systems, or computer program products. Therefore, this application can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this application can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0134] The above description is merely an embodiment of this application and is not intended to limit the scope of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of the claims of this application.
Claims
1. A camera calibration method, wherein, The method includes: Collect multiple point pairs of corresponding 2D points in images captured by a camera and 3D points in a real scene to obtain the first calibration parameter. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points. The second calibration parameters are obtained by iterating the first calibration parameters according to the preset optimization function; Based on the second calibration parameters, determine the mapping relationship table between the 2D points and the 3D points; The step of iterating the first calibration parameters according to a preset optimization function to obtain the second calibration parameters includes: The rotation matrix R from the camera to the ENU and the offset vector t from the origin of the camera coordinate system to the origin of the ENU coordinate system are iterated in the initialization extrinsic parameter matrix [R|t] of the first calibration parameters according to the preset optimization function. Iterate the normal vector n of the road plane in the ENU coordinate system in the first calibration parameters according to the preset optimization function; The optimized and iterated R, t, and n are used as the second calibration parameters; The preset optimization function includes: min (1 / m ∑ (d i / max_d) | Pc - Pt | ) m is the number of pairs of corresponding points of the 2D point and the 3D point, d i is the Euclidean distance of the 3D point to the camera origin, max_d is the Euclidean distance of the point farthest from the camera in the pairs of corresponding points of the 2D point and the 3D point, Pc = [R | t] * Pi, Pc is the 3D point, Pi is the 2D point, and Pt is the point in the ENU coordinate system.
2. The method as described in claim 1, wherein, The first calibration parameters are obtained by acquiring multiple point pairs corresponding to 2D points in images captured by the camera and 3D points in the real scene, including: Based on the PNP algorithm, the initial extrinsic parameter matrix [R|t] and the initial height d from the camera to the ground are calculated. Based on the coordinates of the 2D point, the coordinates of the 3D point in the world coordinate system are calculated according to the initialization extrinsic parameter matrix [R|t]. The initial extrinsic parameter matrix [R|t], the initial height d of the camera from the ground, and the coordinate position of the 3D point in the world coordinate system are used as the first calibration parameters.
3. The method as described in claim 2, wherein, The method further includes: By filtering the coordinates of the 3D points in the world coordinate system and the initial height d of the camera from the ground, the initial normal vector n of the road plane in the ENU coordinate system is determined.
4. The method as described in claim 1, wherein, The preset optimization function further includes: using the optimized and iterated R, t, and n as the second calibration parameters and minimizing the value of the preset optimization function.
5. The method as described in claim 1, wherein, The method further includes: The pose parameters of the roadside camera are obtained based on the mapping table between 2D points and 3D points. Based on the pose parameters of the roadside camera, the latitude, longitude, and height parameters of the target object on the road surface from the perspective of the roadside camera are determined to provide roadside positioning information.
6. A camera calibration device, wherein, The device includes: The acquisition module is used to acquire multiple corresponding point pairs between 2D points in images captured by roadside cameras and 3D points in real roadside scenes to obtain first calibration parameters. The 3D points include coordinate position points in the world coordinate system, and the 2D points include coordinate position points in the pixel coordinate system corresponding to the 3D points. The iteration module is used to iterate the first calibration parameters according to a preset optimization function to obtain the second calibration parameters; The determination module is used to determine the mapping relationship table between the 2D points and the 3D points based on the second calibration parameters; The step of iterating the first calibration parameters according to a preset optimization function to obtain the second calibration parameters includes: The rotation matrix R from the camera to the ENU and the offset vector t from the origin of the camera coordinate system to the origin of the ENU coordinate system are iterated in the initialization extrinsic parameter matrix [R|t] of the first calibration parameters according to the preset optimization function. Iterate the normal vector n of the road plane in the ENU coordinate system in the first calibration parameters according to the preset optimization function; The optimized and iterated R, t, and n are used as the second calibration parameters; The preset optimization function includes: min(1 / m ∑(di / max_d)|Pc - Pt| ) m is the number of corresponding point pairs between the 2D point and the 3D point, di is the Euclidean distance between the 3D point and the camera origin, max_d is the Euclidean distance between the point farthest from the camera among the multiple corresponding point pairs between the 2D point and the 3D point, Pc = [R|t]* Pi, where Pc is the 3D point, Pi is the 2D point, and Pt is the point in the ENU coordinate system.
7. An electronic device, comprising: processor; as well as A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the method of any one of claims 1 to 5.
8. A computer-readable storage medium storing one or more programs, which, when executed by an electronic device including a plurality of applications, cause the electronic device to perform the method of any one of claims 1 to 5.