Method, device, equipment, medium and product for determining quality of calibration parameters

Abstract

This application relates to a method, apparatus, device, medium, and product for determining the quality of calibration parameters. The method includes: acquiring a verification image of a verification pattern captured by a calibrated camera, the calibrated camera comprising multiple lenses, the verification image comprising images captured by each lens; performing stereo matching on the verification image based on the calibration parameters of the calibrated camera to obtain estimated depth information of the bearing plane; determining quality data of a three-dimensional quality index corresponding to the calibration parameters based on the measured depth information and estimated depth information of the bearing plane; and determining the quality assessment result of the calibration parameters based on the quality data of the three-dimensional quality index. This method can more reliably determine the quality of calibration parameters, providing an objective basis for batch quality control of calibration parameters and optimization and improvement of calibration algorithms.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a method, apparatus, device, medium and product for determining calibration parameter quality. Background Technology

[0002] With the rapid development of computer vision technology, camera calibration, as a fundamental and crucial technology in vision systems, has been widely applied in fields such as industrial inspection, autonomous driving, and augmented reality. To determine the mapping relationship between the three-dimensional position of a point in the world coordinate system and its two-dimensional position in the image coordinate system, it is necessary to establish a geometric model of camera imaging. The parameters of the geometric model are the camera parameters, and the process of solving for the camera parameters is called camera calibration.

[0003] During camera calibration, factors such as the number and pose distribution of calibration images, the detection accuracy of feature points, and the selection of initial values ​​for the optimization algorithm can all affect the accuracy of the calibration results, leading to varying degrees of error in the calibration parameters. This, in turn, affects the accuracy and reliability of subsequent vision applications. Therefore, there is an urgent need to provide a reliable method for determining the quality of calibration parameters, providing a basis for quality judgment of calibration parameters, quality control in mass production, and optimization and improvement of calibration algorithms. Summary of the Invention

[0004] Therefore, it is necessary to provide a method, apparatus, equipment, medium, and product for determining the quality of calibration parameters in order to reliably determine the quality of calibration parameters, addressing the aforementioned technical problems.

[0005] Firstly, this application provides a method for determining the quality of calibration parameters, including: Acquire verification images of the verification pattern captured by the calibrated camera; the calibrated camera includes multiple lenses, and the verification images include images captured by each lens separately; the imaging plane of the calibrated camera forms a non-zero angle with the bearing plane of the verification pattern; Based on the calibration parameters of the calibrated camera, stereo matching is performed on the verification image to obtain the estimated depth information of the bearing plane; the estimated depth information is used to indicate the estimated depth value of the physical space point corresponding to each pixel of the verification image on the bearing plane; Based on the measured and estimated depth information of the bearing plane, the quality data of the three-dimensional quality index corresponding to the calibration parameters are determined, and the quality assessment result of the calibration parameters is determined based on the quality data of the three-dimensional quality index. The three-dimensional quality index is used to characterize the ability to reconstruct three-dimensional space using the calibration parameters.

[0006] In one embodiment, based on the measured depth information and estimated depth information of the bearing plane, the quality data of the three-dimensional quality index corresponding to the calibration parameters are determined, including: Based on the estimated depth information, the quality data of the self-quality index corresponding to the calibration parameters are determined; the self-quality index is used to characterize the ability of the calibration parameters to reconstruct depth information. Based on the difference between the measured depth information and the estimated depth information of the bearing plane, the quality data of the accuracy quality index corresponding to the calibration parameters are determined. The quality data of the three-dimensional quality indicators are determined by combining the quality data of the quality indicators themselves, the quality data of the precision quality indicators, and the relative weights of each quality indicator; the relative weights are determined based on the relative importance between the quality indicators.

[0007] In one embodiment, the estimated depth value includes a valid depth value and an invalid depth value; the valid depth value is the estimated depth value of the physical space point corresponding to the successfully matched valid pixel, and the invalid depth value is the estimated depth value of the physical space point corresponding to the unmatched invalid pixel. Based on the estimated depth information, determine the quality data of the self-quality indicators corresponding to the calibration parameters, including: Based on the number of valid pixels corresponding to the effective depth value and the total number of pixels corresponding to the estimated depth value, the quality data of the complete quality index corresponding to the calibration parameters is determined; the complete quality index is used to characterize the coverage completeness of the estimated depth information. Based on the standard deviation of each effective depth value, the quality data of the smoothing quality index corresponding to the calibration parameters are determined; the smoothing quality index is used to characterize the smoothness and stability of the estimated depth information. By combining the quality data of complete quality indicators, the quality data of smoothed quality indicators, and the relative weights of each quality indicator, the quality data of its own quality indicators is determined.

[0008] In one embodiment, the measured depth information includes the measured depth values ​​of multiple preset spatial points on the bearing plane; Based on the difference between the measured depth information and the estimated depth information of the bearing plane, the quality data of the accuracy quality index corresponding to the calibration parameters are determined, including: Based on the difference between the predicted plane obtained by fitting the predicted depth information and the measured plane obtained by fitting the measured depth information, the quality data of the distortion quality index corresponding to the calibration parameters are determined; the distortion quality index is used to characterize the degree of geometric deformation of the predicted depth information. Based on the difference between the estimated depth value at each preset spatial point in the estimated depth information and the measured depth value at each preset spatial point in the measured depth information, the quality data of the accurate quality index corresponding to the calibration parameters are determined; the accurate quality index is used to characterize the accuracy of the estimated depth information relative to the measured depth information. By combining the quality data of the distortion quality index, the quality data of the accurate quality index, and the relative weights of each quality index, the quality data of the precision quality index is determined.

[0009] In one embodiment, determining the quality assessment result of calibration parameters based on quality data of three-dimensional quality indicators includes: By using calibration parameters to project the feature points of the verification pattern onto the predicted projection position and the actual position of the feature points in the verification image, the quality data of the monocular reprojection quality index corresponding to the calibration parameters is determined. The first feature point on the first verification image is projected onto the image plane of the second lens using calibration parameters to obtain the coordinates of the projection point. The quality data of the binocular reprojection quality index corresponding to the calibration parameters is determined by the actual matching coordinates of the first feature point on the second verification image and the coordinates of the projection point. The first verification image is the verification image of the first lens among multiple lenses. The second verification image is the verification image of the second lens. The second feature point on the first verification image is transformed to the epipolar line using calibration parameters. The positional relationship between the actual matching coordinates of the second feature point on the second verification image and the epipolar line is calculated, and the quality data of the epipolar line quality index corresponding to the calibration parameters is determined. The quality assessment results are determined by combining the quality data of monocular reprojection quality index, binocular reprojection quality index, epipolar quality index, and three-dimensional quality index.

[0010] In one embodiment, the quality assessment result is determined by combining the quality data of monocular reprojection quality index, binocular reprojection quality index, epipolar quality index, and 3D quality index, including: Using the quality data from the three-dimensional quality index, the key quality data for calibration parameters are determined; Auxiliary quality data for calibration parameters are determined using quality data from monocular reprojection quality index, binocular reprojection quality index, and epipolar quality index. If the critical quality data is greater than or equal to the first critical threshold, the quality assessment result of the calibration parameters is determined to be Level 1; If the critical quality data is less than the first critical threshold, greater than or equal to the second critical threshold, and the auxiliary quality data is greater than or equal to the target auxiliary threshold, the quality assessment result of the calibration parameters is determined to be level two. If the critical quality data is less than the second critical threshold, or if the critical quality data is less than the first critical threshold, greater than or equal to the second critical threshold, and the auxiliary quality data is less than the target auxiliary threshold, the quality assessment result of the calibration parameters is determined to be level three; level one quality is better than level two quality, and level two quality is better than level three quality.

[0011] Secondly, this application also provides an apparatus for determining the quality of calibration parameters, comprising: The acquisition module is used to acquire the verification image captured by the calibrated camera for the verification pattern; the calibrated camera includes multiple lenses, and the verification image includes the image captured by each lens; the imaging plane of the calibrated camera forms a non-zero angle with the bearing plane of the verification pattern; The estimation module is used to perform stereo matching on the verification image based on the calibration parameters of the calibrated camera to obtain the estimated depth information of the bearing plane; the estimated depth information is used to indicate the estimated depth value of the physical space point corresponding to each pixel of the verification image on the bearing plane; The determination module is used to determine the quality data of the three-dimensional quality index corresponding to the calibration parameters based on the measured depth information and the estimated depth information of the bearing plane, and to determine the quality assessment result of the calibration parameters based on the quality data of the three-dimensional quality index; the three-dimensional quality index is used to characterize the ability to reconstruct three-dimensional space using the calibration parameters.

[0012] Thirdly, this application also provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for determining the quality of calibration parameters provided in the first aspect of this application.

[0013] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the method for determining the quality of calibration parameters provided in the first aspect of this application.

[0014] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the method for determining the quality of calibration parameters provided in the first aspect of this application.

[0015] The aforementioned method, apparatus, equipment, medium, and product for determining the quality of calibration parameters involve acquiring verification images of a verification pattern using a calibrated camera (which includes multiple lenses), and the verification images comprising images acquired by each lens. Based on the calibration parameters of the calibrated camera, stereo matching is performed on the verification images to obtain estimated depth information of the bearing plane. Based on the measured and estimated depth information of the bearing plane, the quality data of the three-dimensional quality indicators corresponding to the calibration parameters are determined, and the quality assessment result of the calibration parameters is determined based on the quality data of the three-dimensional quality indicators. This application determines the quality data of the three-dimensional quality indicators corresponding to the calibration parameters by using the measured and estimated depth information of the bearing plane. It directly reflects the calibration parameters' ability to reconstruct three-dimensional space using depth information, avoiding the potential biases and limitations of two-dimensional evaluation indicators. This provides a more comprehensive and reliable determination of the quality of the calibration parameters, offering an objective basis for batch quality control of calibration parameters and optimization and improvement of calibration algorithms. Meanwhile, by having the imaging plane of the calibrated camera and the bearing plane of the verification pattern form a non-zero angle, the verification image contains perspective distortion information, which enables a more comprehensive evaluation of the stereo matching accuracy and depth reconstruction capability of the calibration parameters under non-parallel viewpoints, further improving the reliability of the quality assessment results. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the drawings used in the description of the embodiments of this application or related technologies will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is an application environment diagram of a method for determining the quality of calibration parameters in one embodiment; Figure 2 This is a flowchart illustrating a method for determining the quality of calibration parameters in one embodiment; Figure 3 This is a flowchart illustrating the process of determining quality data for three-dimensional quality indicators in one embodiment. Figure 4 This is a flowchart illustrating the process of determining quality data for its own quality indicators in one embodiment. Figure 5 This is a flowchart illustrating the process of determining quality data for accuracy quality indicators in one embodiment. Figure 6 This is a flowchart illustrating the process of determining the quality assessment results in one embodiment; Figure 7 This is a structural block diagram of a device for determining the quality of calibration parameters in one embodiment; Figure 8This is an internal structural diagram of a computer device in one embodiment. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0019] It should be noted that the terms "first," "second," etc., used in this application can be used to describe various elements, but these elements are not limited by these terms. These terms are only used to distinguish the first element from the second element. The terms "comprising" and "having," and any variations thereof, used in this application, are intended to cover non-exclusive inclusion. The term "multiple" used in this application refers to two or more. The term "and / or" used in this application refers to one of the embodiments, or any combination of multiple embodiments.

[0020] The method for determining the quality of calibration parameters provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, the calibrated camera 102 can acquire verification images of the verification pattern. These images can then be transmitted to the server 104. The server 104 obtains the verification images and the calibration parameters of the calibrated camera 102. Based on the calibration parameters, it performs stereo matching on the verification images to obtain the estimated depth information of the bearing plane of the calibrated camera 102. Based on the measured depth information and the estimated depth information of the bearing plane, it determines the quality data of the three-dimensional quality indicators corresponding to the calibration parameters, and finally obtains the quality assessment result of the calibration parameters. The server 104 can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0021] It should be noted that the verification image acquired by the calibrated camera 102 can also be transmitted to the terminal. The terminal can then use the verification image and the calibration parameters of the calibrated camera 102 to evaluate the quality of the calibration parameters and obtain a quality evaluation result. The terminal can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can include smart speakers, smart TVs, smart air conditioners, smart in-vehicle devices, etc. Portable wearable devices can include smartwatches, smart bracelets, head-mounted devices, etc. Alternatively, the processor of the calibrated camera 102 can also use the verification image and the calibration parameters of the calibrated camera 102 to obtain the quality evaluation result of the calibration parameters of the calibrated camera 102.

[0022] In one exemplary embodiment, such as Figure 2As shown, a method for determining the quality of calibration parameters is provided, which can be applied to... Figure 1 Taking server 104 as an example, the explanation includes the following steps 201 to 203. Wherein: Step 201: Obtain the verification image captured by the calibrated camera for the verification pattern.

[0023] The calibrated camera includes multiple lenses, and the verification images include images captured by each lens individually. The imaging plane of the calibrated camera forms a non-zero angle with the bearing plane of the verification pattern. The calibrated camera can be a binocular camera, a trinocular camera, or a multi-camera system with more lenses. The bearing plane refers to the physical planar area covered by the verification pattern. This bearing plane corresponds spatially to the physical area of ​​the verification pattern; that is, the verification pattern is attached to or displayed on the bearing plane.

[0024] In this embodiment, a verification platform can be pre-built. For example, a horizontal reference surface is provided, its flatness is calibrated, and a camera fixing mold is set at the center of the horizontal reference surface. The bottom of the mold has suction cups, which firmly fix the calibrated camera in the mold, ensuring that the relative pose between the calibrated camera and the supporting plane of the verification pattern remains stable during verification. A television is placed at a preset angle to the horizontal plane, and a random dot matrix pattern is played on the television screen as the verification pattern. The supporting plane can be the planar area on the television screen displaying the random dot matrix pattern. The random dot matrix pattern provides rich feature points and avoids the influence of periodic patterns on matching accuracy. Each lens in the calibrated camera synchronously captures the random dot matrix pattern displayed on the television screen, acquiring multiple verification images. Each verification image corresponds to the image captured by one lens, and the acquired verification images are uploaded to a server.

[0025] Step 202: Perform stereo matching on the verification image based on the calibration parameters of the calibrated camera to obtain the estimated depth information of the bearing plane.

[0026] The calibration parameters refer to the parameters obtained through the camera calibration process that describe the camera's imaging geometry. For any two lenses in a calibrated camera, stereo matching of two verification images can be performed using their extrinsic parameters (rotation matrix and translation vector) and their respective intrinsic parameters. Predicted depth information can be used to indicate the predicted depth value of physical spatial points on the bearing plane corresponding to each pixel of the verification image. The depth value can refer to the distance from the spatial point along the camera's optical axis to the imaging plane.

[0027] For example, taking a binocular camera, the server can acquire a first verification image from the first lens and a second verification image from the second lens of the calibrated camera, and use the pixels belonging to the verification pattern coverage area in the two images as the objects to be matched. The server can first use the calibration parameters of the calibrated camera to perform epipolar correction on the two images, so that the pixels to be matched in the left and right images are on the same horizontal line. The server uses a stereo matching algorithm (such as semi-global block matching or a stereo matching algorithm based on deep learning) to perform point-by-point matching of the pixels in the verification pattern coverage area. The first verification image can be used as a reference image, and the second verification image can be used as a target image. Using each pixel in the reference image as a reference, matching points are searched on the corresponding epipolar lines of the target image, and the disparity value of each pixel is calculated to obtain a disparity map. Then, combined with the baseline length and focal length in the calibration parameters, the depth value of each pixel is calculated through the triangulation principle to obtain a depth map. The gray value of each effective pixel in the depth map represents the estimated depth value of the physical space point on the bearing plane corresponding to that pixel, thus obtaining the estimated depth information.

[0028] For multi-view camera systems with three or more lenses, the server can select any two lenses from multiple lenses to form a stereo matching pair, calculate the estimated depth information for each lens, and then fuse the multiple estimated depth information into a unified estimated depth information using a fusion strategy (such as averaging, weighted averaging, or median filtering). In one implementation, the server can use a voting mechanism to remove outliers and select the depth value with the highest confidence as the final estimated depth value for that pixel, thereby improving the accuracy and robustness of the estimated depth information.

[0029] Step 203: Based on the measured depth information and estimated depth information of the bearing plane, determine the quality data of the three-dimensional quality index corresponding to the calibration parameters, and determine the quality assessment result of the calibration parameters based on the quality data of the three-dimensional quality index.

[0030] Measured depth information refers to the actual depth values ​​of multiple preset spatial points on the bearing plane, obtained in advance by measuring equipment. For example, multiple ranging points are selected on the bearing plane, and the depth value of each point along the camera's optical axis is measured, i.e., the distance from each point to the camera's imaging plane along the optical axis, thus obtaining the measured depth information for each point. The 3D quality index is used to characterize the ability to reconstruct 3D space using calibration parameters; its quality data is a quantified value calculated from the estimated depth information and the measured depth information.

[0031] For example, the server compares the estimated depth information point-by-point or region-by-region with the pre-measured depth information of the bearing plane. In one possible implementation, the server can determine the quality data of the three-dimensional quality index based on the average absolute error or root mean square error between the estimated depth value and the measured depth value at a preset spatial point, and compare the quality data with a preset quality threshold. For example, if the quality data is less than a first quality threshold, it is judged as excellent; if it is between the first and second quality thresholds, it is judged as qualified; and if it is greater than the second quality threshold, it is judged as unqualified, thereby obtaining the quality assessment result of the calibration parameters.

[0032] In the above method for determining the quality of calibration parameters, verification images are acquired by a calibrated camera targeting a verification pattern. The calibrated camera includes multiple lenses, and the verification images include images acquired by each lens. Stereo matching is performed on the verification images based on the calibration parameters of the calibrated camera to obtain the estimated depth information of the bearing plane. Based on the measured depth information and the estimated depth information of the bearing plane, the quality data of the three-dimensional quality index corresponding to the calibration parameters is determined, and the quality evaluation result of the calibration parameters is determined based on the quality data of the three-dimensional quality index. This embodiment of the application determines the quality data of the three-dimensional quality index corresponding to the calibration parameters by using the measured depth information and the estimated depth information of the bearing plane. It directly reflects the ability of the calibration parameters to reconstruct three-dimensional space using depth information, avoiding the possible biases and limitations of two-dimensional evaluation indicators. This provides a more comprehensive and reliable determination of the quality of the calibration parameters, offering an objective basis for batch quality control of calibration parameters and optimization and improvement of calibration algorithms. Meanwhile, by having the imaging plane of the calibrated camera and the bearing plane of the verification pattern form a non-zero angle, the verification image contains perspective distortion information, which enables a more comprehensive evaluation of the stereo matching accuracy and depth reconstruction capability of the calibration parameters under non-parallel viewpoints, further improving the reliability of the quality assessment results.

[0033] Understandably, the process of determining the quality data of the three-dimensional quality indicators in step 203 is not unique. In order to determine the quality assessment results of the calibration parameters more comprehensively and accurately, the embodiments of this application can divide the quality indicators of the calibration parameters into multiple levels, and use multi-level quality indicators to determine the quality of the calibration parameters more flexibly.

[0034] For example, the server can obtain a preset hierarchical structure corresponding to the quality indicators of the calibration parameters. This preset hierarchical structure includes at least two levels, each level includes at least one set of quality indicators, and each set of quality indicators includes at least one quality indicator. Adjacent levels in the at least two levels are respectively adjacent upper levels and adjacent lower levels. Any upper-level quality indicator in an adjacent upper level corresponds to a set of lower-level quality indicators in an adjacent lower level. The quality data of any upper-level quality indicator in an adjacent upper level can be determined based on the quality data and relative weight of the corresponding lower-level quality indicators. The relative weight can be obtained by performing hierarchical analysis on each quality indicator based on the relative importance among the quality indicators within the same group. Through hierarchical decomposition and layer-by-layer aggregation, the quality assessment problem of the calibration parameters can be transformed into a comprehensive evaluation of multiple quantifiable sub-indicators, making the assessment results more comprehensive and objective.

[0035] In some embodiments, within a preset hierarchical structure, a set of lower-level quality indicators corresponding to the 3D quality index of the calibration parameters may include intrinsic quality indicators and accuracy quality indicators. The intrinsic quality indicators are used to evaluate the calibration parameters from the perspective of their own depth reconstruction capabilities (such as the completeness, smoothness, continuity, and noise of depth information), while the accuracy quality indicators are used to evaluate the consistency between the geometric information reconstructed by the calibration parameters and the real geometric information. By using both intrinsic and accuracy quality indicators, the performance of the calibration parameters under ideal conditions and their absolute accuracy in real-world scenarios can be considered, thereby more accurately and comprehensively determining the quality data of the 3D quality indicators and obtaining more reliable quality evaluation results for the calibration parameters.

[0036] In one exemplary embodiment, such as Figure 3 As shown, the process of determining the quality data of the three-dimensional quality indicators in step 203 includes steps 301 to 303. Wherein: Step 301: Based on the estimated depth information, determine the quality data of the self-quality indicators corresponding to the calibration parameters.

[0037] Among them, the self-quality index is used to characterize the ability of the calibration parameters to reconstruct depth information.

[0038] For example, a set of lower-level quality metrics corresponding to the self-quality metric may include a complete quality metric, a smoothness quality metric, a continuity quality metric, and a noise quality metric. The complete quality metric can be used to reflect the coverage of the estimated depth information, i.e., the proportion of effective depth values ​​in the total number of pixels; the smoothness quality metric can be used to reflect the degree of numerical fluctuation of the estimated depth information in flat areas; the continuity quality metric can be used to reflect whether the depth jumps of the estimated depth information at object edges are continuous and natural; and the noise quality metric can be used to reflect the level of random noise in the estimated depth information. The server can extract feature values ​​corresponding to the above lower-level quality metrics from the estimated depth information, calculate the quality data of each lower-level quality metric, and then combine the relative weights of each lower-level quality metric to obtain the quality data of its own quality metric through weighted summation.

[0039] Step 302: Based on the difference between the measured depth information and the estimated depth information of the bearing plane, determine the quality data of the accuracy quality index corresponding to the calibration parameters.

[0040] Among them, the accuracy quality index can be used to characterize the consistency between the geometric information reconstructed by calibration parameters and the true geometric information. The quality data of the accuracy quality index is a quantitative value obtained by comparing the estimated depth information with the measured depth information.

[0041] For example, a set of lower-level quality indicators corresponding to the accuracy quality indicator may include a distortion quality indicator, an accuracy quality indicator, and a global consistency quality indicator. The distortion quality indicator can be used to reflect the degree of geometric deformation of the estimated depth information relative to the measured depth information in the overall shape. The accuracy quality indicator can be used to reflect the absolute error level between the estimated depth value and the measured depth value at each preset spatial point. The global consistency indicator can be used to reflect the smoothness of the change in depth error across the entire bearing plane. The server can compare the measured depth information and the estimated depth information, calculate the feature values ​​corresponding to the above lower-level quality indicators based on the comparison results, obtain the quality data of each lower-level quality indicator, and then combine the relative weights of each lower-level quality indicator to obtain the quality data of the accuracy quality indicator through weighted summation.

[0042] Step 303: Combine the quality data of its own quality indicators, the quality data of precision quality indicators, and the relative weights of each quality indicator to determine the quality data of the three-dimensional quality indicators.

[0043] The relative weights are determined based on the relative importance of the quality indicators.

[0044] For example, the server can obtain the preset relative importance between its own quality indicators and the accuracy quality indicators, determine the relative weight of its own quality indicators and the relative weight of the accuracy quality indicators based on the relative importance, normalize and update the quality data of its own quality indicators and the quality data of the accuracy quality indicators, and then combine the quality data of its own quality indicators, the relative weight of its own quality indicators, the quality data of the accuracy quality indicators, and the relative weight of the accuracy quality indicators to calculate the quality data of the three-dimensional quality indicators.

[0045] In this embodiment, the three-dimensional quality index is decomposed into an intrinsic quality index that characterizes the inherent reconstruction capability and an accuracy quality index that characterizes the absolute geometric accuracy. The two indices are calculated based on the estimated depth information itself and the difference between the estimated depth information and the measured depth information. The indices are then weighted and fused together with their relative weights. This achieves a comprehensive and hierarchical quantitative evaluation of the three-dimensional reconstruction performance of the calibration parameters, avoiding the one-sidedness of evaluation based on a single index.

[0046] Taking a set of lower-level quality indicators corresponding to its own quality indicators, including complete quality indicators and smoothed quality indicators, as an example, this paper introduces the process of determining the quality data of its own quality indicators.

[0047] Due to insufficient texture information within the verification pattern coverage area, pixels located at image boundaries, or being occluded, there may be pixels that fail to match during stereo matching. These pixels correspond to invalid depth values ​​in the depth map. The server can record the positions of invalid pixels by marking or masking, and the estimated depth information can include the positions of these invalid pixels.

[0048] In an exemplary embodiment, the estimated depth value may include a valid depth value and an invalid depth value; the valid depth value is the estimated depth value of the physical spatial point corresponding to a successfully matched valid pixel, and the invalid depth value is the estimated depth value of the physical spatial point corresponding to an unmatched invalid pixel, such as... Figure 4 As shown, step 301 includes steps 401 to 403. Wherein: Step 401: Based on the number of valid pixels corresponding to the effective depth value and the total number of pixels corresponding to the estimated depth value, determine the quality data of the complete quality index corresponding to the calibration parameters.

[0049] Among them, the completeness quality index is used to characterize the degree of coverage completeness of the estimated depth information.

[0050] For example, the server can count the number of valid depth values ​​in the estimated depth information, obtain the total number of pixels in the reference verification image that belong to the area covered by the verification pattern, divide the number of valid pixels by the total number of pixels to obtain the integrity rate, and use the integrity rate as the quality data of the integrity quality index.

[0051] Step 402: Based on the standard deviation of each effective depth value, determine the quality data of the smoothing quality index corresponding to the calibration parameters.

[0052] Among them, the smoothing quality index is used to characterize the smoothness and stability of the estimated depth information.

[0053] For example, the server extracts all valid depth values ​​and calculates the standard deviation of the valid depth values. In one implementation, the server may also calculate the standard deviation only for valid depth values ​​within a flat region with uniform texture on the bearing plane to avoid interference from normal depth jumps at object edges, and use the calculated standard deviation as quality data for the smoothing quality index.

[0054] Step 403: Combine the quality data of complete quality indicators, the quality data of smoothed quality indicators, and the relative weights of each quality indicator to determine the quality data of its own quality indicators.

[0055] For example, the server can normalize the quality data of the complete quality indicators and the quality data of the smoothed quality indicators before performing weighted fusion.

[0056] In this embodiment, by decomposing its own quality index into a complete quality index representing the integrity of coverage and a smooth quality index representing the smoothness of numerical values, and calculating them based on the effective pixel ratio and the standard deviation of the effective depth value respectively, and then combining them with weights for weighted fusion, a refined evaluation of the intrinsic quality of the estimated depth information is achieved, providing a reliable foundation for the comprehensive calculation of subsequent three-dimensional quality indices.

[0057] Taking a set of lower-level quality indicators corresponding to the accuracy quality indicator, including distortion quality indicators and accuracy quality indicators, as an example, this paper introduces the process of determining the quality data of the accuracy quality indicator.

[0058] In one exemplary embodiment, the measured depth information includes the measured depth values ​​of multiple preset spatial points on the bearing plane, such as... Figure 5 As shown, step 302 includes steps 501 to 503. Wherein: Step 501: Based on the difference between the predicted plane obtained by fitting the predicted depth information and the measured plane obtained by fitting the measured depth information, determine the quality data of the distortion quality index corresponding to the calibration parameters.

[0059] Among them, the distortion quality index is used to characterize the degree of geometric deformation of the estimated depth information. The estimated plane can refer to the geometric shape of the bearing plane reconstructed by the estimated depth information and calibration parameters, while the measured plane can refer to the geometric shape of the bearing plane that is actually measured.

[0060] For example, during the measurement phase, the three-dimensional coordinates (X, Y, Z) of each preset spatial point can be obtained using a high-precision measuring device, where Z is the depth value. Then, the equation of the measured plane is obtained by fitting the three-dimensional coordinates. The server, based on the pixel coordinates of each preset spatial point in the verification image, combined with the depth value of the corresponding pixel in the estimated depth information and the intrinsic parameters of the calibrated camera (such as focal length and principal point coordinates), obtains the estimated three-dimensional coordinates of each preset spatial point through back projection. Then, the estimated plane is obtained by fitting the estimated three-dimensional coordinates. The server can calculate the normal vector of the measured plane and the normal vector of the estimated plane, determine the angle between the two normal vectors or the average distance deviation between the two planes, and use this angle or deviation as quality data for the distortion quality index.

[0061] Step 502: Based on the difference between the estimated depth value at each preset spatial point in the estimated depth information and the measured depth value at each preset spatial point in the measured depth information, determine the quality data of the accurate quality index corresponding to the calibration parameters.

[0062] Among them, the accuracy quality index is used to characterize the accuracy of the estimated depth information relative to the measured depth information.

[0063] For example, the server can obtain the estimated depth value of each preset spatial point in the estimated depth information and the measured depth value in the measured depth information, calculate the absolute or relative depth error of each point, and take the average or root mean square value of the errors of all points as the quality data of the accurate quality indicator.

[0064] Step 503: Combine the quality data of the distortion quality index, the quality data of the accurate quality index, and the relative weights of each quality index to determine the quality data of the precision quality index.

[0065] For example, the server can normalize the quality data of distorted quality indicators and the quality data of accurate quality indicators before performing weighted fusion.

[0066] In this embodiment, the accuracy quality index is decomposed into a distortion quality index that characterizes the degree of geometric deformation and an accuracy quality index that characterizes the absolute depth accuracy. The two indices are calculated based on the differences in plane fitting and the differences in depth point by point, respectively. Then, they are combined with weights for weighted fusion to achieve a refined evaluation of the geometric reconstruction accuracy of the calibration parameters, providing a reliable accuracy basis for the comprehensive calculation of the three-dimensional quality index.

[0067] To more comprehensively determine the quality of calibration parameters, embodiments of this application may also combine two-dimensional quality indices based on reprojection and epipolar geometric constraints to comprehensively determine the performance of calibration parameters.

[0068] In one exemplary embodiment, such as Figure 6As shown, the process of determining the quality assessment results of the calibration parameters in step 203 includes steps 601 to 604. Wherein: Step 601: Using calibration parameters, estimate the projection position of the feature points of the verification pattern and the actual position of the feature points in the verification image, and determine the quality data of the monocular reprojection quality index corresponding to the calibration parameters.

[0069] Among them, the monocular reprojection quality index can be used to determine the accuracy of individual lens calibration parameters (such as intrinsic parameters and distortion coefficients).

[0070] For example, the server extracts feature points of the verification pattern from the verification image, obtains their three-dimensional coordinates in the calibration pattern coordinate system, projects these three-dimensional points onto the image plane of the lens using the calibration parameters of the calibrated camera, obtains the estimated projection coordinates, calculates the Euclidean distance between the estimated projection coordinates and the actual detection coordinates, and takes the average value of all feature points as the quality data of the monocular reprojection quality index.

[0071] Step 602: Project the first feature point on the first verification image onto the image plane of the second lens using the calibration parameters to obtain the coordinates of the projection point. Determine the quality data of the binocular reprojection quality index corresponding to the calibration parameters by using the actual matching coordinates of the first feature point on the second verification image and the coordinates of the projection point.

[0072] The first verification image is the verification image of the first shot out of multiple shots, and the second verification image is the verification image of the second shot. The actual matching coordinates can refer to the actual pixel position of the physical spatial point corresponding to the first feature point on the second verification image.

[0073] For example, the server detects a first feature point in the first verification image and obtains its actual matching coordinates in the second verification image. Using calibration parameters (including intrinsic parameters of the two lenses, distortion coefficients, and extrinsic parameters between them), the feature point is projected onto the image plane of the second lens to obtain the coordinates of the projected point. The Euclidean distance between the projected point coordinates and the actual matching coordinates is calculated, and the average value of all matching feature point pairs is taken as the quality data of the binocular reprojection quality index.

[0074] Step 603: Transform the second feature point on the first verification image to the epipolar line using calibration parameters, calculate the positional relationship between the actual matching coordinates of the second feature point on the second verification image and the epipolar line, and determine the quality data of the epipolar line quality index corresponding to the calibration parameters.

[0075] For example, the server selects a second feature point on the first verification image, calculates the corresponding epipolar equation in the second verification image according to the calibration parameters, obtains the actual matching coordinates of the second feature point in the second verification image, calculates the vertical distance from the matching coordinates to the epipolar line, and takes the average value of all feature point pairs as the quality data of the epipolar line quality index.

[0076] Step 604: Combine the quality data of monocular reprojection quality index, binocular reprojection quality index, epipolar quality index, and three-dimensional quality index to determine the quality assessment result.

[0077] In one possible implementation, the server can use a weighted summation method, combining preset weights for each quality indicator with quality data to obtain comprehensive quality data. The server can also employ fuzzy comprehensive evaluation or machine learning models to perform nonlinear combination or pattern recognition on multi-dimensional quality data, thereby outputting quality assessment results. Furthermore, the quality assessment results can be compared with preset performance thresholds, security baselines, or user expectations to determine whether specific quality standards have been met.

[0078] In other implementations, the server can use the quality data of the three-dimensional quality index to determine the key quality data of the calibration parameters; and use the quality data of the monocular reprojection quality index, the binocular reprojection quality index, and the epipolar quality index to determine the auxiliary quality data of the calibration parameters.

[0079] For example, the server can directly identify the quality data of 3D quality indicators as key quality data. For auxiliary quality data, the server can perform a weighted summation or average of the quality data of monocular reprojection quality indicators, binocular reprojection quality indicators, and epipolar quality indicators, using the result as auxiliary quality data. For instance, the weights of each 2D quality indicator can be pre-set based on the sensitivity of projection accuracy and epipolar constraints according to the actual application scenario. When the accuracy of stereo matching has high requirements for epipolar constraints, the weight of the epipolar quality indicator can be increased.

[0080] If the critical quality data is greater than or equal to the first critical threshold, the quality assessment result of the calibration parameters is determined to be Level 1.

[0081] If the critical quality data is less than the first critical threshold, greater than or equal to the second critical threshold, and the auxiliary quality data is greater than or equal to the target auxiliary threshold, the quality assessment result of the calibration parameters is determined to be level two.

[0082] If the critical quality data is less than the second critical threshold, or if the critical quality data is less than the first critical threshold, greater than or equal to the second critical threshold, and the auxiliary quality data is less than the target auxiliary threshold, the quality assessment result of the calibration parameters is determined to be level three.

[0083] Among them, the quality of Grade 1 is better than that of Grade 2, and the quality of Grade 2 is better than that of Grade 3.

[0084] In this embodiment, the accuracy of three-dimensional reconstruction is used as the key judgment criterion, while the accuracy of two-dimensional reprojection and epipolar constraint are used as auxiliary correction conditions. This takes into account the supplementary role of two-dimensional indicators, avoids misjudgment that may be caused by relying on a single indicator, and makes the quality assessment results more reasonable and reliable.

[0085] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages in other steps. It is understood that the steps in different embodiments can be freely combined as needed, and all non-contradictory solutions formed by such combinations are within the scope of protection of this application.

[0086] Based on the same inventive concept, this application also provides an apparatus for determining the calibration parameter quality to implement the method for determining the calibration parameter quality described above. The solution provided by this apparatus is similar to the solution described in the above method. Therefore, the specific limitations in one or more embodiments of the apparatus for determining the calibration parameter quality provided below can be found in the limitations of the method for determining the calibration parameter quality described above, and will not be repeated here.

[0087] In one exemplary embodiment, such as Figure 7 As shown, a device for determining the quality of calibration parameters is provided, comprising: an acquisition module 701, an estimation module 702, and a determination module 703, wherein: The acquisition module 701 is used to acquire the verification image captured by the calibrated camera for the verification pattern; the calibrated camera includes multiple lenses, and the verification image includes the image captured by each lens respectively; the imaging plane of the calibrated camera and the bearing plane of the verification pattern form a non-zero angle.

[0088] The estimation module 702 is used to perform stereo matching on the verification image based on the calibration parameters of the calibrated camera to obtain the estimated depth information of the bearing plane; the estimated depth information is used to indicate the estimated depth value of the physical space point corresponding to each pixel of the verification image on the bearing plane.

[0089] The determination module 703 is used to determine the quality data of the three-dimensional quality index corresponding to the calibration parameters based on the measured depth information and the estimated depth information of the bearing plane, and to determine the quality assessment result of the calibration parameters based on the quality data of the three-dimensional quality index; the three-dimensional quality index is used to characterize the ability to reconstruct three-dimensional space using the calibration parameters.

[0090] Each module in the aforementioned device for determining the quality of calibration parameters can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device, or stored in the memory of a computer device as software, so that the processor can call and execute the corresponding operations of each module.

[0091] In one exemplary embodiment, a computer device is provided, which may be a server, and its internal structure diagram may be as follows: Figure 8 As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and databases. The internal memory provides the environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communication with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for determining the quality of calibration parameters.

[0092] Those skilled in the art will understand that Figure 8 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0093] In one exemplary embodiment, a computer device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method for determining the quality of calibration parameters provided in the first aspect of the present application.

[0094] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the steps of the method for determining the quality of calibration parameters provided in the first aspect of the embodiments of this application.

[0095] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps of the method for determining the quality of calibration parameters provided in the first aspect of the embodiments of this application.

[0096] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0097] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, artificial intelligence (AI) processors, etc., and are not limited to these.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0099] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A method for determining the quality of calibration parameters, characterized in that, The method includes: Acquire a verification image captured by a calibrated camera for a verification pattern; the calibrated camera includes multiple lenses, and the verification image includes an image captured by each of the lenses respectively; the imaging plane of the calibrated camera forms a non-zero angle with the bearing plane of the verification pattern; Based on the calibration parameters of the calibrated camera, stereo matching is performed on the verification image to obtain the estimated depth information of the bearing plane; the estimated depth information is used to indicate the estimated depth value of the physical space point corresponding to each pixel of the verification image on the bearing plane; Based on the measured depth information and the estimated depth information of the bearing plane, the quality data of the three-dimensional quality index corresponding to the calibration parameter is determined, and the quality assessment result of the calibration parameter is determined based on the quality data of the three-dimensional quality index; the three-dimensional quality index is used to characterize the ability to reconstruct three-dimensional space using the calibration parameter.

2. The method according to claim 1, characterized in that, The process of determining the quality data of the three-dimensional quality index corresponding to the calibration parameters based on the measured depth information and the estimated depth information of the bearing plane includes: Based on the estimated depth information, the quality data of the self-quality index corresponding to the calibration parameter is determined; the self-quality index is used to characterize the ability of the calibration parameter to reconstruct depth information; Based on the difference between the measured depth information and the estimated depth information of the bearing plane, the quality data of the accuracy quality index corresponding to the calibration parameter is determined; The quality data of the three-dimensional quality indicators are determined by combining the quality data of the self-quality indicators, the quality data of the precision quality indicators, and the relative weights of each quality indicator; the relative weights are determined based on the relative importance between the quality indicators.

3. The method according to claim 2, characterized in that, The estimated depth value includes a valid depth value and an invalid depth value; the valid depth value is the estimated depth value of the physical space point corresponding to the successfully matched valid pixel point, and the invalid depth value is the estimated depth value of the physical space point corresponding to the unmatched invalid pixel point. The step of determining the quality data of the self-quality index corresponding to the calibration parameters based on the estimated depth information includes: Based on the number of valid pixels corresponding to the effective depth value and the total number of pixels corresponding to the estimated depth value, the quality data of the complete quality index corresponding to the calibration parameter is determined; the complete quality index is used to characterize the coverage completeness of the estimated depth information. Based on the standard deviation of each effective depth value, the quality data of the smoothing quality index corresponding to the calibration parameter is determined; the smoothing quality index is used to characterize the smoothness and stability of the estimated depth information. By combining the quality data of the complete quality indicators, the quality data of the smoothed quality indicators, and the relative weights of each quality indicator, the quality data of the self-quality indicator is determined.

4. The method according to claim 2, characterized in that, The measured depth information includes the measured depth values ​​of multiple preset spatial points on the bearing plane; The quality data for determining the accuracy quality index corresponding to the calibration parameters based on the difference between the measured depth information and the estimated depth information of the bearing plane includes: Based on the difference between the predicted plane obtained by fitting the predicted depth information and the measured plane obtained by fitting the measured depth information, the quality data of the distortion quality index corresponding to the calibration parameter is determined; the distortion quality index is used to characterize the degree of geometric deformation of the predicted depth information. Based on the difference between the estimated depth value at each preset spatial point in the estimated depth information and the measured depth value at each preset spatial point in the measured depth information, the quality data of the accurate quality index corresponding to the calibration parameter is determined; the accurate quality index is used to characterize the accuracy of the estimated depth information relative to the measured depth information. The quality data of the precision quality index is determined by combining the quality data of the distortion quality index, the quality data of the accurate quality index, and the relative weights of each quality index.

5. The method according to claim 1, characterized in that, The quality assessment result of determining the calibration parameter based on the quality data of the three-dimensional quality indicators includes: The quality data of the monocular reprojection quality index corresponding to the calibration parameters are determined by using the estimated projection position of the feature points of the verification pattern and the actual position of the feature points in the verification image. The first feature point on the first verification image is projected onto the image plane of the second lens using the calibration parameters to obtain the coordinates of the projection point. The quality data of the binocular reprojection quality index corresponding to the calibration parameters is determined by the actual matching coordinates of the first feature point on the second verification image and the coordinates of the projection point. The first verification image is the verification image of the first lens among multiple lenses. The second verification image is the verification image of the second lens. The second feature point on the first verification image is transformed to the epipolar line using the calibration parameters. The positional relationship between the actual matching coordinates of the second feature point on the second verification image and the epipolar line is calculated. The quality data of the epipolar line quality index corresponding to the calibration parameters is then determined. The quality assessment result is determined by combining the quality data of the monocular reprojection quality index, the quality data of the binocular reprojection quality index, the quality data of the epipolar quality index, and the quality data of the three-dimensional quality index.

6. The method according to claim 5, characterized in that, The process of determining the quality assessment result by combining the quality data of the monocular reprojection quality index, the binocular reprojection quality index, the epipolar quality index, and the three-dimensional quality index includes: Using the quality data of the three-dimensional quality indicators, the key quality data of the calibration parameters are determined; Using the quality data of the monocular reprojection quality index, the quality data of the binocular reprojection quality index, and the quality data of the epipolar quality index, auxiliary quality data for the calibration parameters are determined. If the key quality data is greater than or equal to the first key threshold, the quality assessment result of the calibration parameter is determined to be level one; If the key quality data is less than the first key threshold, greater than or equal to the second key threshold, and the auxiliary quality data is greater than or equal to the target auxiliary threshold, the quality assessment result of the calibration parameter is determined to be level two. If the key quality data is less than the second key threshold, or if the key quality data is less than the first key threshold, greater than or equal to the second key threshold, and the auxiliary quality data is less than the target auxiliary threshold, the quality assessment result of the calibration parameter is determined to be level three; the quality of level one is better than the quality of level two, and the quality of level two is better than the quality of level three.

7. A device for determining the quality of calibration parameters, characterized in that, The device includes: An acquisition module is used to acquire a verification image captured by a calibrated camera for a verification pattern; the calibrated camera includes multiple lenses, and the verification image includes an image captured by each of the lenses respectively; the imaging plane of the calibrated camera forms a non-zero angle with the bearing plane of the verification pattern; The estimation module is used to perform stereo matching on the verification image based on the calibration parameters of the calibrated camera to obtain the estimated depth information of the bearing plane; the estimated depth information is used to indicate the estimated depth value of the physical space point on the bearing plane corresponding to each pixel of the verification image; The determination module is used to determine the quality data of the three-dimensional quality index corresponding to the calibration parameter based on the measured depth information and the estimated depth information of the bearing plane, and to determine the quality assessment result of the calibration parameter based on the quality data of the three-dimensional quality index; the three-dimensional quality index is used to characterize the ability to reconstruct three-dimensional space using the calibration parameter.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images