Joint calibration method, device and medium for multi-camera extrinsic parameters

Abstract

This application discloses a method, apparatus, device, and medium for joint calibration of multi-camera extrinsic parameters, which can be used in the field of autonomous driving. The specific implementation scheme is as follows: First, determine the overlapping region of the viewpoints of each camera, and obtain a 2D verification point set from the image of the overlapping region, wherein the 2D verification point set includes at least one verification point; Second, perform single-camera extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera; Third, calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera, and calculate a loss function based on the coordinates of the 3D point, wherein the loss function is used to measure the superposition error of the calibration of each camera under the current extrinsic parameters; Fourth, perform joint calibration based on the loss function to obtain the target extrinsic parameters of each camera. This application embodiment uses a loss function to measure the superposition error of the calibration of each camera under the current extrinsic parameters and performs joint calibration based on the loss function, thereby eliminating the superposition of errors in multi-camera calibration.

Description

[0001] This application is a divisional application of patent application number 201910955707.0 (the original application was filed on October 9, 2019, and the invention was entitled "Joint Calibration Method, Apparatus, Device and Medium for Multiple Camera Extrinsic Parameters"). Technical Field

[0002] This application relates to image processing technology, which can be used in the field of autonomous driving, and particularly to a camera extrinsic parameter calibration technology, specifically to a method, apparatus, device and medium for joint calibration of multiple camera extrinsic parameters. Background Technology

[0003] Camera extrinsic calibration refers to obtaining the transformation matrix (composed of rotation matrix and translation vector) from the world coordinate system to the camera coordinate system after the camera is installed. This transformation matrix is ​​the camera extrinsic parameter. Camera extrinsic parameters are essential for connecting the real-world position of an object to its pixel position in an image. They are crucial in fields requiring precise object location within an image, such as autonomous driving and security applications. Therefore, calculating accurate camera extrinsic parameters is essential. Summary of the Invention

[0004] This application provides a method, apparatus, device, and medium for joint calibration of extrinsic parameters of multiple cameras, so as to reduce the superposition error of extrinsic parameters of multiple cameras.

[0005] In a first aspect, embodiments of this application provide a method for joint calibration of multiple camera extrinsic parameters, including:

[0006] Determine the overlapping area of ​​each camera's viewpoint, and obtain a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point;

[0007] Perform individual extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera;

[0008] The coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and a loss function is calculated based on the coordinates of the 3D point. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters.

[0009] Joint calibration is performed based on the loss function to obtain the target extrinsic parameters of each camera.

[0010] One embodiment of the above application has the following advantages or beneficial effects: by adding a verification step on the basis of the prior art, the loss function is calculated based on the 3D points mapped back to the world coordinate system from the verification points. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters, thereby playing a certain verification role. Based on the loss function, the joint calibration of multiple cameras is performed to determine the final target extrinsic parameters, thereby largely eliminating the superposition of multi-camera errors and providing a good foundation for subsequent multi-camera fusion.

[0011] Optionally, the joint calibration based on the loss function to obtain the target extrinsic parameters of each camera includes:

[0012] Repeat the following operation until the number of executions meets the preset threshold, and use the camera extrinsic parameters corresponding to the loss functions whose values ​​meet the preset conditions as the target extrinsic parameters for each camera:

[0013] Perform extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera;

[0014] The coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and the loss function is calculated based on the coordinates of the 3D point.

[0015] One embodiment of the above application has the following advantages or beneficial effects: by iterating multiple times to obtain the loss function for each iteration and obtaining the optimal solution from it, the optimal camera extrinsic parameters are obtained.

[0016] Optionally, calculating the loss function based on the coordinates of the 3D point includes:

[0017] Take any verification point as the current verification point, and calculate the variance based on the coordinates of the 3D point of the current verification point obtained for each camera.

[0018] The variances calculated for each validation point are summed, and the sum is used as the loss function.

[0019] One embodiment of the above application has the following advantages or beneficial effects: when the true value of the verification point in the world coordinate system is unknown, the variance is calculated to measure the degree of convergence of the 3D points mapped by each camera to the same verification point in the world coordinate system. The better the convergence, the more the reprojection error of each camera tends to be in the same direction, and the smaller the superposition error. This plays the role of measuring and verifying the superposition error of each camera under the current extrinsic parameters through the loss function.

[0020] Optionally, calculating the loss function based on the coordinates of the 3D point includes:

[0021] Take any verification point as the current verification point, and determine the distance between the 3D point of the current verification point calculated for each camera and the true value of the current verification point in the world coordinate system, and calculate the average value of the distance.

[0022] Based on the true value, the variance is calculated according to the coordinates of the 3D point of the current verification point obtained for each camera;

[0023] The variance and mean calculated for the current verification point are weighted and summed.

[0024] The weighted summation results obtained from each verification point are summed, and the sum is used as the loss function.

[0025] One embodiment of the above application has the following advantages or beneficial effects: when the true value of the verification point in the world coordinate system can be obtained, the mean and variance are used together as part of the loss function. The variance can measure the degree of clustering of each 3D point, while the mean can measure the distance of each 3D point from the true value. Furthermore, the superposition error of each camera calibration is measured from the two perspectives of the degree of clustering and the distance from the true value, so as to improve the accuracy of finding the optimal solution of the extrinsic parameters.

[0026] Optionally, calculating the loss function based on the coordinates of the 3D point further includes:

[0027] Determine the minimum distance between any two 3D points among the 3D points of the current verification point calculated for each camera;

[0028] Based on the calculated loss function, the distance value calculated for each verification point is added to the loss function.

[0029] One embodiment of the above application has the following advantages or beneficial effects: based on the loss function calculated by variance and mean, an additional dimension is added, namely the distance between any two 3D points. This is because the smaller the distance between any two 3D points, the smaller the superposition error of the extrinsic parameters can be indicated from this dimension, thereby further improving the accuracy of measuring superposition error by loss function.

[0030] Optionally, the step of performing single-camera extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera includes:

[0031] Using the nearest point matching method and the method of minimizing reprojection error, the extrinsic parameters of each camera are calibrated to obtain the current extrinsic parameters of each camera.

[0032] Secondly, embodiments of this application also provide a joint calibration device for multiple camera extrinsic parameters, comprising:

[0033] The verification point set acquisition module is used to determine the overlapping area of ​​each camera viewpoint and acquire a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point.

[0034] The single-camera extrinsic calibration module is used to calibrate the extrinsic parameters of each camera individually, and obtain the current extrinsic parameters of each camera.

[0035] The loss function calculation module is used to calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera, and to calculate the loss function based on the coordinates of the 3D point. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters.

[0036] The multi-camera joint calibration module is used to perform joint calibration based on the loss function to obtain the target extrinsic parameters of each camera.

[0037] Thirdly, embodiments of this application also provide an electronic device, including:

[0038] At least one processor; and

[0039] A memory communicatively connected to the at least one processor; wherein,

[0040] The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the joint calibration method for multi-camera extrinsic parameters as described in any embodiment of this application.

[0041] Fourthly, embodiments of this application also provide a non-transitory computer-readable storage medium storing computer instructions, the computer instructions being used to cause the computer to execute the joint calibration method for multi-camera extrinsic parameters described in any embodiment of this application.

[0042] One embodiment of the above application has the following advantages or beneficial effects: by adding a verification step on the basis of the prior art, the optimal external parameters of the joint camera calibration are determined by verifying the results of the 3D return of the point set, which solves the limitation of only considering the reprojection error of a single camera in the previous single camera calibration, and largely eliminates the superposition of multi-camera errors, providing a good foundation for subsequent multi-camera fusion.

[0043] Other effects of the above-mentioned alternative methods will be described below in conjunction with specific embodiments. Attached Figure Description

[0044] The accompanying drawings are provided for a better understanding of this solution and do not constitute a limitation of this application. Wherein:

[0045] Figure 1 This is a flowchart illustrating the joint calibration method for multi-camera extrinsic parameters according to the first embodiment of this application;

[0046] Figure 2 This is a schematic diagram of the superposition error of multi-camera calibration in the first embodiment of this application;

[0047] Figure 3 This is a flowchart illustrating the joint calibration method for multi-camera extrinsic parameters according to the second embodiment of this application;

[0048] Figure 4 This is a schematic diagram of the structure of a multi-camera extrinsic parameter joint calibration device according to the third embodiment of this application;

[0049] Figure 5 This is a block diagram of an electronic device used to implement the joint calibration method for multi-camera extrinsic parameters according to the embodiments of this application. Detailed Implementation

[0050] The following description, in conjunction with the accompanying drawings, illustrates exemplary embodiments of this application, including various details to aid understanding. These should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this application. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0051] In scenarios involving the calculation of camera extrinsic parameters, some applications require distributed multi-camera setups capturing the same scene from different positions and angles. Examples include roadside sensing, where multiple cameras are distributed at intersections, and subsequent multi-camera fusion is performed for target localization in that scenario. However, calibrating the extrinsic parameters of a single camera introduces errors. Therefore, during distributed multi-camera calibration, the maximum errors in the calibration of extrinsic parameters from different cameras can accumulate, leading to inaccurate 3D target position and impacting subsequent multi-camera fusion. This application's embodiment addresses this limitation by adding a verification step to the existing technology. By verifying the 3D results from the verification point set, the optimal extrinsic parameters for joint camera calibration are determined. This overcomes the limitation of previous single-camera calibration methods that only considered single-camera reprojection errors, largely eliminating the accumulation of multi-camera errors and providing a solid foundation for subsequent multi-camera fusion.

[0052] Figure 1 This is a flowchart illustrating a method for joint calibration of multi-camera extrinsic parameters according to the first embodiment of this application. This embodiment is applicable to situations requiring joint calibration of multi-camera extrinsic parameters, such as multiple cameras distributed in a road test perception application scenario. This method can be executed by a joint calibration device for multi-camera extrinsic parameters, implemented in software and / or hardware, preferably configured in an electronic device, such as a computer or server. Figure 1 As shown, the method specifically includes the following:

[0053] S101. Determine the overlapping area of ​​each camera's viewpoint, and obtain a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point.

[0054] In distributed multi-camera scenarios, such as road testing and perception applications, multiple cameras are typically used to simultaneously capture images of the same location or target, such as an intersection. Therefore, there will be overlapping areas in the viewpoints of each camera—areas that each camera can capture. The selected 2D verification point can be one point within this overlapping area, or multiple points, preferably points that did not participate in the camera extrinsic parameter calibration in S102, as this is essential for the verification purpose.

[0055] S102. Perform single-camera extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera.

[0056] The extrinsic parameters of a single camera can be calibrated using any method available in the prior art, such as nearest-point matching combined with a method to minimize the reprojection error. Alternatively, specific markers or additional sensors can be used to assist in the calibration process. This application does not impose any limitations on these methods.

[0057] S103. Calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera, and calculate the loss function based on the coordinates of the 3D point. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters.

[0058] S104. Perform joint calibration based on the loss function to obtain the target extrinsic parameters of each camera.

[0059] Taking the method of nearest-neighbor matching combined with minimizing reprojection error (BA) for extrinsic parameter calibration as an example, due to the uncertainty of nearest-neighbor matching, the results of multiple executions of nearest-neighbor matching and BA have a certain degree of randomness. For example, for the same target point on the camera image, if the 3D point corresponding to the target point in the world coordinate system is calculated based on the extrinsic parameters obtained from each calibration, such as... Figure 2 As shown in (a), point A is the ground truth point of the target point, and the other points are the 3D points calculated for each calibration. It can be observed that, with the same error, these 3D points may appear at different positions of the ground truth point A. It should also be noted that in the process of calculating the 3D points of the verification points in the world coordinate system based on the extrinsic parameters, it is also necessary to combine the ground equation and the camera's intrinsic parameters. However, in the embodiments of this application, the ground equation and the camera's intrinsic parameters can be considered as fixed ground truth values.

[0060] Furthermore, such as Figure 2 As shown in (b), point A is the ground truth point of the target point. Taking the calibration of four cameras as an example, the other four points are the 3D points of the target point in the world coordinate system calculated based on the extrinsic parameters of each of the four cameras. It can be seen from (b) that when the extrinsic parameters of the four cameras are obtained independently, although the error of each camera meets the requirements, the maximum error will be superimposed because the error directions of each camera are different, as shown by the dashed line in (b). The length of the dashed line can be used to measure the magnitude of the superimposed error.

[0061] Therefore, in this embodiment, after calculating the extrinsic parameters of a single camera, a verification step is added. That is, the coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and a loss function is calculated based on the coordinates of these 3D points. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters; that is, the smaller the loss function, the smaller the superposition error. For example, the loss function is used to verify whether the error between the 3D point of the verification point back to the world coordinate system and the true value of the verification point under the currently calculated extrinsic parameters meets the requirements. Furthermore, in the joint calibration of multi-camera extrinsic parameters, the 3D point corresponding to the verification point calculated for each camera can be obtained. Then, the loss function can be determined by calculating the mean, variance, or distance from the true value of these 3D points, thereby measuring the superposition error in the joint calibration process of multi-camera extrinsic parameters, and thus determining whether the current extrinsic parameters of each camera meet the requirements, or by finding the global optimal solution of the multi-camera joint calibration extrinsic parameters through multiple iterations, thereby greatly reducing the superposition error of extrinsic parameters.

[0062] As one implementation method for calculating the loss function, the loss function is calculated based on the coordinates of the 3D points, including:

[0063] Take any verification point as the current verification point, and calculate the variance based on the coordinates of the 3D point of the current verification point obtained for each camera.

[0064] The variances calculated for each validation point are summed, and the sum is used as the loss function.

[0065] If the set of verification points includes only one verification point, the coordinates of the 3D point of the verification point in the world coordinate system can be calculated directly based on the extrinsic parameters of each camera, and the variance can be calculated based on the coordinates of each 3D point to obtain the loss function.

[0066] If the set of validation points includes more than two validation points, then calculations need to be performed separately for each validation point. Specifically, any validation point is taken as the current validation point, and then the coordinates of the current validation point in the 3D coordinate system are calculated based on the extrinsic parameters of each camera. The variance is then calculated based on the coordinates of each 3D point. After calculating for each validation point in this way, the variance calculated for each validation point is determined, and these variances are summed to obtain the loss function.

[0067] It should be noted here that by calculating variance, the degree of clustering of 3D points mapped by each camera to the same verification point in the world coordinate system can be measured. Figure 2 As can be seen from (c) in the figure, the better the convergence, the more the reprojection errors of each camera tend to be in the same direction, and the smaller the superposition error (as shown by the dotted line in the figure). This plays the role of measuring and verifying the superposition error of each camera under the current extrinsic parameters through the loss function. That is, the smaller the loss function, the smaller the superposition error, and the optimal the corresponding extrinsic parameters.

[0068] Furthermore, the calculation of variance can include two scenarios. When the true value of the verification point in the world coordinate system can be obtained, this true value can be used as the expected value of each corresponding 3D point, and then the variance can be calculated based on the distance between each 3D point and this true value. When the true value of the verification point cannot be obtained, the mean of each 3D point in the x, y, and z directions can be calculated first to obtain the optimal estimate of the true value, and then the variance can be calculated based on the distance between each 3D point and this optimal estimate.

[0069] As another implementation of calculating the loss function, the loss function is calculated based on the coordinates of the 3D points, including:

[0070] Take any verification point as the current verification point, and determine the distance between the 3D point of the current verification point calculated for each camera and the true value of the current verification point in the world coordinate system, and calculate the average value of the distance.

[0071] Based on the true value, the variance is calculated according to the coordinates of the 3D point of the current verification point obtained for each camera;

[0072] The variance and mean calculated for the current verification point are weighted and summed.

[0073] The weighted summation results obtained from each verification point are summed, and the sum is used as the loss function.

[0074] In this implementation, which is a case where the true value of the current verification point in the world coordinate system can be obtained, the distance between each 3D point and the true value can be calculated first, and the mean of the distance can be calculated. Then, similarly, the true value is used as the expected value of each 3D point corresponding to the current verification point. The variance is then calculated based on the distance between each 3D point and the true value. The variance and mean calculated for the current verification point are then summed in a weighted manner. Finally, the weighted sums calculated for each verification point are summed, and the sum is used as the loss function.

[0075] It should be noted that, when the true values ​​of the verification points in the world coordinate system can be obtained, the mean and variance are used together as part of the loss function. The variance can measure the degree of clustering of each 3D point, while the mean can measure the distance of each 3D point from the true value. By setting the weights, the importance of the variance and mean to the superposition error can be set, thereby further measuring the superposition error of each camera calibration from the two perspectives of clustering degree and distance from the true value, so as to improve the accuracy of finding the optimal solution of the extrinsic parameters.

[0076] Furthermore, in another embodiment, calculating the loss function based on the coordinates of the 3D points further includes:

[0077] Determine the minimum distance between any two 3D points among the 3D points of the current verification point calculated for each camera;

[0078] Based on the calculated loss function, the distance value calculated for each verification point is added to the loss function.

[0079] Specifically, in this implementation, another dimension is added to the already calculated loss function: the minimum distance between any two 3D points. Since the distance between two points can also measure the magnitude of the stacking error, including this dimension in the loss function allows for further measurement of the stacking error from the perspective of its magnitude. Similarly, the smaller the loss function, the smaller the stacking error, and the optimal the corresponding camera extrinsic parameters. Furthermore, weights can be assigned to the variance, mean, and distance dimensions, and the final sum can be obtained as the loss function through a weighted summation.

[0080] The technical solution of this application embodiment adds a verification step to the existing technology. The loss function is calculated based on the 3D points mapped back to the world coordinate system from the verification points. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters. When the loss function is minimized, the error direction difference of the current extrinsic parameters of each camera can be minimized, which greatly reduces the superposition phenomenon of errors and reduces the superposition error. Thus, the joint calibration of multiple cameras is performed based on the loss function to determine the final target extrinsic parameters, providing a good foundation for subsequent multi-camera fusion.

[0081] Figure 3 This is a flowchart illustrating the joint calibration method for multi-camera extrinsic parameters according to the second embodiment of this application. This embodiment is further optimized based on the above embodiment. Figure 3 As shown, the method specifically includes the following:

[0082] S201. Determine the overlapping area of ​​each camera's viewpoint, and obtain a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point.

[0083] S202. Using the nearest point matching method and the method of minimizing reprojection error, extrinsic parameters are calibrated for each camera to obtain the current extrinsic parameters of each camera.

[0084] S203. Calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera, and calculate the loss function based on the coordinates of the 3D point.

[0085] S204. Determine and save the loss function with the minimum current value. Check whether the current number of executions meets the preset threshold. If not, return to execute S202. If it does, execute S205.

[0086] S205. Use the camera extrinsic parameters corresponding to the currently saved loss function as the target extrinsic parameters for each camera.

[0087] In this embodiment, the loss function with the smallest value is selected from the loss functions obtained by multiple iterations of the camera extrinsic parameters, and the camera extrinsic parameters corresponding to the loss function with the smallest value are taken as the optimal solution, i.e., the target extrinsic parameters.

[0088] Specifically, the number of iterations can be preset. For example, in the first iteration, the extrinsic parameters of each camera are calibrated, and the loss function for this iteration is calculated based on these parameters. In the second iteration, the extrinsic parameters of each camera are calibrated, and the loss function for this iteration is calculated based on these parameters. Then, the two loss functions from the first and second iterations are compared to see which one has the smallest value, and this loss function with the smallest value is saved. Then, in the third iteration, the loss function calculated for the third iteration is compared with the loss function saved in the previous iteration, and the loss function with the smallest value is saved. The above operation is repeated until the number of iterations meets the threshold, and the extrinsic parameters corresponding to the loss function saved at this time are taken as the optimal solution.

[0089] The technical solution of this application embodiment adds a verification and optimization step to the existing technology. It calculates the loss function based on the 3D points mapped back to the world coordinate system from the verification points. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters. Based on the loss function obtained by multiple iterations, the optimal solution is selected by comparison to find the global optimal solution of the multi-camera joint calibration extrinsic parameters. The direction of the multi-camera reprojection error is constrained to the same direction, thereby greatly reducing the superposition error of extrinsic parameters and obtaining the optimal target extrinsic parameters, which provides a good foundation for subsequent multi-camera fusion.

[0090] Figure 4 This is a schematic diagram of a multi-camera extrinsic parameter joint calibration device according to a third embodiment of this application. This embodiment is applicable to situations requiring joint calibration of multi-camera extrinsic parameters, such as multiple cameras distributed in a road test perception application scenario. This device can implement the multi-camera extrinsic parameter joint calibration method described in any embodiment of this application. Figure 4 As shown, the device 300 specifically includes:

[0091] The verification point set acquisition module 301 is used to determine the overlapping area of ​​each camera viewpoint and acquire a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point.

[0092] The single-camera extrinsic calibration module 302 is used to calibrate the extrinsic parameters of each camera separately to obtain the current extrinsic parameters of each camera;

[0093] The loss function calculation module 303 is used to calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera, and to calculate the loss function based on the coordinates of the 3D point. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters.

[0094] The multi-camera joint calibration module 304 is used to perform joint calibration based on the loss function to obtain the target extrinsic parameters of each camera.

[0095] Optionally, the multi-camera joint calibration module 304 is specifically used for:

[0096] Repeat the following operation until the number of executions meets the preset threshold, and use the camera extrinsic parameters corresponding to the loss functions whose values ​​meet the preset conditions as the target extrinsic parameters for each camera:

[0097] Perform extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera;

[0098] The coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and the loss function is calculated based on the coordinates of the 3D point.

[0099] Optionally, the loss function calculation module 303 includes:

[0100] The 3D point coordinate calculation unit is used to calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera.

[0101] The first loss function calculation unit is specifically used for:

[0102] Take any verification point as the current verification point, calculate the variance based on the coordinates of the 3D point of the current verification point obtained for each camera; sum the variances obtained for each verification point, and use the sum as the loss function.

[0103] Optionally, the loss function calculation module 303 includes:

[0104] The 3D point coordinate calculation unit is used to calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera.

[0105] The second loss function calculation unit is specifically used for:

[0106] Take any verification point as the current verification point, and determine the distance between the 3D point of the current verification point calculated for each camera and the true value of the current verification point in the world coordinate system, and calculate the average value of the distance.

[0107] Based on the true value, the variance is calculated according to the coordinates of the 3D point of the current verification point obtained for each camera;

[0108] The variance and mean calculated for the current verification point are weighted and summed.

[0109] The weighted summation results obtained from each verification point are summed, and the sum is used as the loss function.

[0110] Optionally, the loss function calculation module further includes a third loss function calculation unit, specifically used for:

[0111] Determine the minimum distance between any two 3D points among the 3D points of the current verification point calculated for each camera;

[0112] Based on the loss function calculated by the first loss function calculation unit or the second loss function calculation unit, the distance value calculated for each verification point is added to the loss function.

[0113] Optionally, the single-camera extrinsic calibration module 302 is specifically used for:

[0114] Using the nearest point matching method and the method of minimizing reprojection error, the extrinsic parameters of each camera are calibrated to obtain the current extrinsic parameters of each camera.

[0115] The multi-camera extrinsic parameter joint calibration device 300 provided in this application embodiment can execute the multi-camera extrinsic parameter joint calibration method provided in any embodiment of this application, and has the corresponding functional modules and beneficial effects of the method execution. Content not described in detail in this embodiment can be referred to the description in any method embodiment of this application.

[0116] According to embodiments of this application, this application also provides an electronic device and a readable storage medium.

[0117] like Figure 5 The diagram shown is a block diagram of an electronic device for a joint calibration method of multi-camera extrinsic parameters according to an embodiment of this application. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present application described and / or claimed herein.

[0118] like Figure 5As shown, the electronic device includes one or more processors 401, a memory 402, and interfaces for connecting the components, including high-speed interfaces and low-speed interfaces. The components are interconnected via different buses and can be mounted on a common motherboard or otherwise as required. The processors can process instructions executed within the electronic device, including instructions stored in or on memory to display graphical information of a GUI on an external input / output device (such as a display device coupled to the interface). In other embodiments, multiple processors and / or multiple buses can be used with multiple memories and multiple memory modules, if desired. Similarly, multiple electronic devices can be connected, each providing some of the necessary operations (e.g., as a server array, a group of blade servers, or a multiprocessor system). Figure 5 Take a processor 401 as an example.

[0119] The memory 402 is the non-transitory computer-readable storage medium provided in this application. The memory stores instructions executable by at least one processor to cause the at least one processor to perform the joint calibration method for multi-camera extrinsic parameters provided in this application. The non-transitory computer-readable storage medium of this application stores computer instructions for causing a computer to perform the joint calibration method for multi-camera extrinsic parameters provided in this application.

[0120] Memory 402, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the multi-camera extrinsic parameter joint calibration method in the embodiments of this application (e.g., attached...). Figure 4 The verification point set acquisition module 301, single-camera extrinsic parameter calibration module 302, loss function calculation module 303, and multi-camera joint calibration module 304 are shown. The processor 401 executes various server functions and data processing by running non-transient software programs, instructions, and modules stored in the memory 402, thereby implementing the multi-camera extrinsic parameter joint calibration method in the above method embodiment.

[0121] The memory 402 may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function. The data storage area may store data created by the use of the electronic device implementing the joint calibration method for multi-camera extrinsics according to the embodiments of this application. Furthermore, the memory 402 may include high-speed random access memory and may also include non-transient memory, such as at least one disk storage device, flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory 402 may optionally include memory remotely located relative to the processor 401. These remote memories can be connected via a network to the electronic device implementing the joint calibration method for multi-camera extrinsics according to the embodiments of this application. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0122] The electronic device implementing the multi-camera extrinsic parameter joint calibration method of the embodiments of this application may further include: an input device 403 and an output device 404. The processor 401, memory 402, input device 403, and output device 404 may be connected via a bus or other means. Figure 4 Taking the example of a connection between China and Israel via a bus.

[0123] Input device 403 can receive input digital or character information, as well as key signal input related to user settings and function control of the electronic device implementing the multi-camera extrinsic parameter joint calibration method of the embodiments of this application. Examples of input devices include touchscreens, keypads, mice, trackpads, touchpads, joysticks, one or more mouse buttons, trackballs, and joysticks. Output device 404 may include display devices, auxiliary lighting devices (e.g., LEDs), and haptic feedback devices (e.g., vibration motors). The display device may include, but is not limited to, liquid crystal displays (LCDs), light-emitting diode (LED) displays, and plasma displays. In some embodiments, the display device may be a touchscreen.

[0124] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, application-specific integrated circuits (ASICs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0125] These computational programs (also referred to as programs, software, software applications, or code) include machine instructions for a programmable processor and can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. As used herein, the terms “machine-readable medium” and “computer-readable medium” refer to any computer program product, device, and / or apparatus (e.g., disk, optical disk, memory, programmable logic device (PLD)) used to provide machine instructions and / or data to a programmable processor, including machine-readable media that receive machine instructions as machine-readable signals. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor.

[0126] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0127] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0128] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0129] According to the technical solution of this application embodiment, by adding a verification step on the basis of the prior art, a loss function is calculated based on the 3D points mapped back to the world coordinate system from the verification points. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters, thereby playing a certain verification role. Based on the loss function, the joint calibration of multiple cameras is performed to determine the final target extrinsic parameters, thereby largely eliminating the superposition of multi-camera errors and providing a good foundation for subsequent multi-camera fusion.

[0130] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this application can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this application can be achieved, and this is not limited herein.

[0131] The specific embodiments described above do not constitute a limitation on the scope of protection of this application. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for joint calibration of extrinsic parameters of multiple cameras, characterized in that, include: Determine the overlapping area of ​​each camera's viewpoint, and obtain a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point; For each camera, perform single-camera extrinsic parameter calibration to obtain the current extrinsic parameters of each camera. The verification point did not participate in the single-camera extrinsic parameter calibration. The coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and the loss function is determined by calculating the variance of the coordinates of the 3D point. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters. Joint calibration is performed based on the loss function to obtain the target extrinsic parameters of each camera.

2. The method according to claim 1, characterized in that, The joint calibration based on the loss function yields the target extrinsic parameters for each camera, including: Repeat the following operation until the number of executions meets the preset threshold, and use the camera extrinsic parameters corresponding to the loss functions whose values ​​meet the preset conditions as the target extrinsic parameters for each camera: Perform extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera; The coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and the loss function is calculated based on the coordinates of the 3D point.

3. The method of claim 1, wherein, The step of determining the loss function by calculating the variance of the coordinates of the 3D points includes: Take any verification point as the current verification point, and calculate the variance based on the coordinates of the 3D point of the current verification point obtained for each camera. The variances calculated for each validation point are summed, and the sum is used as the loss function.

4. The method according to claim 3 or 1, characterized in that, Also includes: Determine the minimum distance between any two 3D points among the 3D points of the current verification point calculated for each camera; Based on the calculated loss function, the distance value calculated for each verification point is added to the loss function.

5. The method according to claim 1 or 2, characterized in that, The step of performing individual camera extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera includes: Using the nearest point matching method and the method of minimizing reprojection error, the extrinsic parameters of each camera are calibrated to obtain the current extrinsic parameters of each camera.

6. A joint calibration device for multiple camera extrinsic parameters, characterized in that, include: The verification point set acquisition module is used to determine the overlapping area of ​​each camera viewpoint and acquire a 2D verification point set in the image of the overlapping area, wherein the 2D verification point set includes at least one verification point. The single-camera extrinsic calibration module is used to perform single-camera extrinsic calibration for each camera to obtain the current extrinsic parameters of each camera. The verification point does not participate in the single-camera extrinsic calibration. The loss function calculation module is used to calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera, and to determine the loss function by calculating the variance of the coordinates of the 3D point. The loss function is used to measure the superposition error of each camera calibration under the current extrinsic parameters. The multi-camera joint calibration module is used to perform joint calibration based on the loss function to obtain the target extrinsic parameters of each camera.

7. The apparatus according to claim 6, characterized in that, The multi-camera joint calibration module is specifically used for: Repeat the following operation until the number of executions meets the preset threshold, and use the camera extrinsic parameters corresponding to the loss functions whose values ​​meet the preset conditions as the target extrinsic parameters for each camera: Perform extrinsic parameter calibration for each camera to obtain the current extrinsic parameters of each camera; The coordinates of the 3D point corresponding to each verification point in the world coordinate system are calculated using the current extrinsic parameters of each camera, and the loss function is calculated based on the coordinates of the 3D point.

8. The apparatus of claim 6, wherein, The loss function calculation module includes: The 3D point coordinate calculation unit is used to calculate the coordinates of the 3D point corresponding to each verification point in the world coordinate system using the current extrinsic parameters of each camera. The first loss function calculation unit is specifically used for: Take any verification point as the current verification point, calculate the variance based on the coordinates of the 3D point of the current verification point obtained for each camera; sum the variances obtained for each verification point, and use the sum as the loss function.

9. The apparatus of claim 8 or 6, wherein, The loss function calculation module further includes a third loss function calculation unit, specifically used for: Determine the minimum distance between any two 3D points among the 3D points of the current verification point calculated for each camera; Based on the calculated loss function, the distance value calculated for each verification point is added to the loss function.

10. The apparatus of claim 6 or 7, wherein, The single-camera extrinsic parameter calibration module is specifically used for: Using the nearest point matching method and the method of minimizing reprojection error, the extrinsic parameters of each camera are calibrated to obtain the current extrinsic parameters of each camera.

11. An electronic device, comprising: include: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform the joint calibration method for multi-camera extrinsics according to any one of claims 1-5.

12. A non-transitory computer-readable storage medium having stored thereon computer instructions, wherein, The computer instructions are used to cause the computer to execute the joint calibration method for multi-camera extrinsic parameters as described in any one of claims 1-5.

Images