Binocular depth calibration method, computer equipment and readable storage medium
By learning the second-order depth calibration coefficient in binocular depth perception technology and calibrating the depth information using a high confidence point cloud, the problem of inaccurate depth perception in the prior art is solved and the calibration accuracy is improved.
Patent Information
- Application Number
- CN202411984324.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-12-27
- Publication Date
- 2025-04-29
AI Technical Summary
The existing binocular depth perception technology is inaccurate in real applications, mainly due to the error introduced by the parallax to depth conversion.
The network-aware depth value is calibrated by learning the second-order depth calibration coefficient based on high confidence point clouds. The specific steps include: acquiring the parallax map and trusted point cloud according to the input image, obtaining the sample depth values in the depth map respectively, constructing a calibration function based on the proportional relationship between the parallax error, the parameter error in the camera and the preset depth value, solving the calibration coefficients through the least squares method, and performing binocular depth calibration.
The accuracy of binocular depth calibration is improved, and the impact of parallax and in-camera parameter error on depth perception is reduced by calibrating depth information using a trusted point cloud with high confidence.
Smart Images
Figure CN119906812A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of depth calibration, and in particular to a binocular depth calibration method, a computer device and a readable storage medium. Background Art
[0002] The core idea of binocular depth perception technology is to use two cameras at a certain distance to capture the left and right perspectives of the same scene, and calculate the depth of the object based on the principle of parallax. However, the conversion from parallax to depth will introduce additional errors, resulting in inaccurate depth perception in actual applications. Summary of the invention
[0003] In view of this, the purpose of the present invention is to overcome the shortcomings of the prior art and provide a binocular depth calibration method, device, computer equipment and readable storage medium for learning second-order depth calibration coefficients based on high-confidence point clouds to calibrate the depth values perceived by the network.
[0004] The present invention provides the following technical solutions:
[0005] In a first aspect, the present invention provides a binocular depth calibration method, comprising:
[0006] Acquire a first depth map according to a disparity map corresponding to an input image, and acquire a second depth map according to a credible point cloud corresponding to the input image;
[0007] Acquire a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively;
[0008] Constructing a calibration function based on the proportional relationship between the parallax error, the camera intrinsic parameter error and the preset depth value;
[0009] Calculating the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient;
[0010] Perform binocular depth calibration according to the calibration coefficient.
[0011] In one embodiment, the constructing of a calibration function based on a proportional relationship between a parallax error, an intrinsic camera parameter error, and a preset depth value comprises:
[0012] Acquire a conversion relationship between the preset depth value, the preset disparity value and a preset camera intrinsic parameter, wherein the conversion relationship includes: a quotient of the preset camera intrinsic parameter and the disparity value is the preset depth value;
[0013] Acquire a first proportional relationship between the preset depth value and the parallax error according to the conversion relationship;
[0014] Acquire a second proportional relationship between the preset depth value and the camera intrinsic parameter error according to the conversion relationship;
[0015] The calibration function is determined according to the first proportional relationship and the second proportional relationship.
[0016] In one embodiment, calculating the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient includes:
[0017] Determine a first matrix according to the first sampling depth value, determine a sample vector according to each of the second sampling depth values, and determine a vector to be solved according to the parameter to be solved in the calibration function;
[0018] Solving the vector to be solved according to the first and the sample vectors based on the least squares method to obtain a parameter value to be solved;
[0019] The calibration coefficient is obtained according to the parameter value to be solved.
[0020] In one implementation, solving the vector to be solved based on the first matrix and the sample vector based on the least squares method to obtain the parameter value to be solved includes:
[0021] determining a first product of the first matrix and the sample vector;
[0022] Determine the difference between the vector to be solved and the first product to obtain an error vector; and determine the sum of squares of the error vectors;
[0023] Derivative the square sum with respect to the vector to be solved, and set the derivative equal to a target value to obtain a derivative equation;
[0024] The derivative equation is solved to obtain the parameter value to be solved.
[0025] In one embodiment, performing binocular depth calibration according to the calibration coefficient includes:
[0026] Calculating a calibration error based on the calibration coefficient;
[0027] Determining whether the calibration coefficient meets a condition according to a preset error threshold and the calibration error;
[0028] If the calibration coefficient meets the conditions, binocular depth calibration is performed according to the calibration coefficient;
[0029] If the calibration coefficient does not meet the condition, the step of acquiring the coordinates of a plurality of sampling points according to the input image is performed again.
[0030] In one embodiment, judging whether the calibration coefficient meets a condition according to a preset error threshold and the calibration error includes:
[0031] Determining whether the calibration error is less than the preset error threshold;
[0032] If the calibration error is less than the preset error threshold, the calibration coefficient satisfies the condition;
[0033] If the calibration error is greater than or equal to the preset error threshold, the calibration coefficient does not meet the condition.
[0034] In one embodiment, the acquiring a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively includes:
[0035] Acquire coordinates of multiple sampling points according to the input image;
[0036] Acquire a first sampling depth value corresponding to the coordinates of each sampling point from the first depth map;
[0037] The second sampling depth value corresponding to each of the sampling point coordinates is obtained from the second depth map.
[0038] In one implementation, the acquiring of a plurality of sampling point coordinates according to the input image comprises:
[0039] Performing instance segmentation on the input image to obtain multiple segmentation masks;
[0040] The coordinates of each sampling point are extracted from each segmentation mask.
[0041] In a second aspect, the present invention proposes a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the binocular depth calibration method as described in the first aspect is implemented.
[0042] In a third aspect, the present invention proposes a computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the binocular depth calibration method as described in the first aspect is implemented.
[0043] The binocular depth calibration method, computer device and readable storage medium disclosed in the present invention obtain a first depth map according to the disparity map corresponding to the input image, and obtain a second depth map according to the credible point cloud corresponding to the input image; obtain multiple first sampling depth values and multiple second sampling depth values from the first depth map and the second depth map respectively; construct a calibration function based on the proportional relationship between the disparity error, the camera internal parameter error and the preset depth value; calculate the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient; and perform binocular depth calibration according to the calibration coefficient. In this way, a calibration function is constructed according to the relationship between the depth value and the disparity error and the camera internal parameter error respectively, and the calibration function is further solved using a credible point cloud with high confidence, and binocular depth calibration is performed according to the calibration coefficient obtained by the solution to improve the calibration accuracy. BRIEF DESCRIPTION OF THE DRAWINGS
[0044] In order to more clearly illustrate the technical solution of the present invention, the following is a brief introduction to the drawings required for use in the embodiments. It should be understood that the following drawings only illustrate certain embodiments of the present invention and should not be regarded as limiting the scope of protection of the present invention. In each of the drawings, similar components are numbered similarly.
[0045] Figure 1 A schematic diagram of a process of the binocular depth calibration method proposed in this embodiment is shown;
[0046] Figure 2 Another schematic diagram of the process of binocular depth calibration method proposed in this embodiment is shown;
[0047] Figure 3 Another schematic diagram of the process of the binocular depth calibration method proposed in this embodiment is shown;
[0048] Figure 4 Another schematic diagram of the process of the binocular depth calibration method proposed in this embodiment is shown;
[0049] Figure 5 Another schematic diagram of the process of the binocular depth calibration method proposed in this embodiment is shown;
[0050] Figure 6 A schematic diagram of the structure of the binocular depth calibration device proposed in this embodiment is shown.
[0051] Description of the accompanying drawings:
[0052] 600-binocular depth calibration device; 601-acquisition module; 602-sampling module; 603-construction module; 604-solution module; 605-calibration module. DETAILED DESCRIPTION
[0053] The technical solutions in the embodiments of the present invention will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments.
[0054] The components of the embodiments of the present invention generally described and shown in the drawings herein may be arranged and designed in a variety of different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.
[0055] Hereinafter, the terms "including", "having" and their cognates, which may be used in various embodiments of the present invention, are intended only to indicate specific features, numbers, steps, operations, elements, components or combinations of the foregoing items, and should not be understood as first excluding the existence of one or more other features, numbers, steps, operations, elements, components or combinations of the foregoing items or adding the possibility of one or more features, numbers, steps, operations, elements, components or combinations of the foregoing items.
[0056] Furthermore, the terms “first”, “second”, “third”, etc. are merely used for distinguishing descriptions and are not to be understood as indicating or implying relative importance.
[0057] Unless otherwise defined, all terms (including technical terms and scientific terms) used herein have the same meanings as those generally understood by those skilled in the art to which the various embodiments of the present invention belong. The terms (such as those defined in generally used dictionaries) will be interpreted as having the same meanings as the contextual meanings in the relevant technical field and will not be interpreted as having idealized meanings or overly formal meanings unless clearly defined in the various embodiments of the present invention.
[0058] Example 1
[0059] Currently, the depth values perceived by the binocular network trained based on public datasets are not accurate. There are two reasons for this: first, the parallax is inversely proportional to the depth, which makes the network insensitive to the depth error of distant objects; second, the parallax distribution and camera intrinsic parameters of different public datasets are different.
[0060] The disclosed embodiments provide a binocular depth calibration method for learning a second-order depth calibration coefficient based on a high-confidence point cloud to calibrate a network-perceived depth value.
[0061] See also Figure 1 A binocular depth calibration method includes steps S101 to S105, and each step is described in detail below.
[0062] Step S101 : acquiring a first depth map according to a disparity map corresponding to an input image, and acquiring a second depth map according to a credible point cloud corresponding to the input image.
[0063] In this embodiment, the input image is input into a common binocular network to obtain a disparity map corresponding to the input image, and the disparity map is converted using camera intrinsic parameters to obtain a first depth map. Common binocular networks include igev-stereo, psmnet, etc.
[0064] At the same time, the input point cloud is obtained by inputting the image, and the input point cloud is the depth calibration true value of the input image; the input point cloud is further preliminarily filtered by the point cloud filtering algorithm, and repeated scanning and comparison are performed to obtain a point cloud with a higher confidence level, that is, a credible point cloud; the credible point cloud is further calculated using the external parameters of the lidar-camera to obtain a second depth map projected into the camera image coordinate system.
[0065] It should be noted that since parallax may introduce errors to the depth, and the three-dimensional coordinate information in the trusted point cloud directly reflects the true shape and structure of the object, the depth value in the second depth map is usually more reliable than the depth value in the first depth map.
[0066] Step S102: Acquire a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively.
[0067] In this embodiment, a plurality of first sampling depth values are randomly obtained from the first depth map, and a plurality of second sampling depth values are randomly obtained from the second depth map. The coordinates of each first sampling depth value should correspond to the coordinates of each second sampling depth value, so as to serve as the data basis for obtaining binocular depth calibration.
[0068] See also Figure 2 In a specific embodiment, step S102 includes steps S1021 to S1023, and each step is described in detail below.
[0069] Step S1021, obtaining coordinates of multiple sampling points according to the input image.
[0070] In this embodiment, a plurality of sampling points are randomly acquired from the input image, and the coordinates of the sampling points are recorded for subsequent acquisition of depth information from the depth map.
[0071] Step S1022: Acquire a first sampling depth value corresponding to each sampling point coordinate from the first depth map.
[0072] In this embodiment, for each sampling point, the corresponding pixel is found in the first depth map according to the sampling point coordinates on the input image, and the depth value of the pixel is read. The depth value is the first sampling depth value of the sampling point in the first depth map.
[0073] Step S1023: Acquire a second sampling depth value corresponding to each sampling point coordinate from the second depth map.
[0074] In this embodiment, for each sampling point, the corresponding pixel is found in the second depth map according to the sampling point coordinates on the input image, and the depth value of the pixel is read. The depth value is the second sampling depth value of the sampling point in the second depth map.
[0075] In a specific embodiment, step S1021 includes: performing instance segmentation on the input image to obtain multiple segmentation masks; and extracting the coordinates of each sampling point in each segmentation mask.
[0076] In this embodiment, an instance segmentation network is used to perform instance segmentation on the input image to obtain segmentation masks of different objects in the input image for subsequent depth pair sampling.
[0077] Furthermore, in the segmentation masks of different objects or backgrounds in instance segmentation, multiple sampling points are randomly obtained, and the position coordinates of the sampling points are recorded to obtain multiple sampling point coordinates. The mask-based sampling method can obtain more diverse sampling points and prevent the sampling from being concentrated in a certain area, resulting in data redundancy.
[0078] Step S103: constructing a calibration function based on the proportional relationship between the parallax error, the camera intrinsic parameter error and the preset depth value.
[0079] In this embodiment, since the depth value is determined according to the quotient of the camera internal parameters and the disparity value, and the binocular network is not sensitive enough to the depth error of distant objects, and the disparity distribution and camera internal parameters of different public data sets are different, a calibration function can be constructed according to the proportional relationship between the disparity error, the camera internal parameter error and the preset depth value.
[0080] See also Figure 3 In a specific embodiment, step S103 includes steps S1031 to S1034, and each step is described in detail below.
[0081] Step S1031, obtaining a conversion relationship between the preset depth value, the preset disparity value and the preset camera intrinsic parameter.
[0082] In this embodiment, the quotient of the preset camera intrinsic parameter and the disparity value is taken as the preset depth value. The conversion relationship between the preset disparity value, the preset disparity value and the preset camera intrinsic parameter is shown in Formula 1:
[0083]
[0084] In the formula, Z0 is the preset parallax value; D is the preset parallax value; A is the preset camera internal parameter; e is the baseline length, which is the distance between the optical centers of the two binocular cameras; f is the camera focal length. e f, A is the simplified form of ef.
[0085] Step S1032: acquiring a first proportional relationship between the preset depth value and the parallax error according to the conversion relationship.
[0086] In this embodiment, it can be known from formula (1) that Z0 is proportional to 1 / D. After differentiating both sides, it can be known that dZ0 is proportional to (1 / D 2 )×dD. Since Z0 is proportional to 1 / D, 1 / D2 is equivalent to Therefore dZ0 is proportional to That is, the first proportional relationship is
[0087] Step S1033: acquiring a second proportional relationship between the preset depth value and the camera intrinsic parameter error according to the conversion relationship.
[0088] In this embodiment, it can be known from formula (1) that Z0 is proportional to A, that is, dZ0 is proportional to dA. Therefore, the second proportional relationship is δZ0∝δA.
[0089] Step S1034: determining the calibration function according to the first proportional relationship and the second proportional relationship.
[0090] In this embodiment, the errors caused by the parallax and the camera intrinsic parameters should have a second-order relationship with the depth itself, so the optimal calibration coefficient formula should be a second-order equation. Adding a first-order relationship can make the error correction smoother. The calibration function formula is shown in formula (2):
[0091]
[0092] Among them, Z' is the depth value after calibration, Z is the depth value before calibration, α is the calibration coefficient, a, b, c are the parameters to be solved.
[0093] Step S104: calculating the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient.
[0094] In this embodiment, the calibration function is constrained by obtaining the first sampling depth value Z' and the second sampling depth value Z* to solve the parameters a, b, and c of the calibration coefficient α, thereby obtaining the calibration coefficient α.
[0095] See also Figure 4 In a specific embodiment, step S104 includes steps S1041 to S1042, and each step is described in detail below.
[0096] Step S1041, determining a first matrix according to the first sampling depth value, determining a sample vector according to each of the second sampling depth values, and determining a vector to be solved according to the parameters to be solved in the calibration function.
[0097] In this embodiment, since the second-order equation of a single variable is solved, at least three sample points are required, namely (Z'1, Z1*), (Z'2, Z2*), (Z'3, Z3*). A Vandermonde matrix V is constructed based on the sample data, and its form is:
[0098]
[0099] Each second sampling depth value constitutes a sample vector A:
[0100]
[0101] The various parameters to be solved form the vector B to be solved:
[0102] B=[cba] T
[0103] There is the following relationship between the Vandermonde matrix V, the sample vector A, and the vector to be solved B:
[0104] A=VB
[0105] Step S1042: Solve the vector to be solved based on the first matrix and the sample vector based on the least squares method to obtain a parameter value to be solved.
[0106] In this embodiment, the least square method is used to solve the vector B to be solved according to the Vandermonde matrix V and the sample vector A to obtain the parameter value to be solved in the vector to be solved: B = (V T V) -1 V T A.
[0107] In a specific embodiment, step S1042 includes: determining a first product of the first matrix and the sample vector; determining a difference between the vector to be solved and the first product to obtain an error vector; and determining the sum of squares of the error vectors; differentiating the sum of squares with respect to the vector to be solved, and setting the derivative equal to a target value to obtain a derivative equation; solving the derivative equation to obtain the parameter value to be solved.
[0108] In this embodiment, the least squares method is a mathematical optimization technique that finds the best function match for the data by minimizing the sum of squared errors.
[0109] First, the first product of the first matrix V and the sample vector A is determined; then the difference between the vector B to be solved and the first product is determined to obtain the error vector E, that is, E=A-VB.
[0110] Furthermore, the sum of squares of the error vector E is calculated 2 , that is, E 2 =(A-VB) T (A-VB) = A T AA T VB-B T V T A+B T V T VB. Further, in order to find the 2 The smallest B needs to be E 2 Taking the derivative about B and setting the derivative equal to zero, we get the derivative equation:
[0111] Furthermore, the data of the first matrix and the data of the sample vector are substituted into the derivative equation for solving to obtain the parameter value to be solved, wherein the expression of the vector to be solved is B=(V T V) -1 V T A.
[0112] Step S1043, obtaining the calibration coefficient according to the parameter value to be solved.
[0113] In this embodiment, after the parameter value to be solved is determined, the parameter value to be solved is substituted into α=aZ in the calibration function. 2 +bZ+c, the specific expression of the calibration coefficient α can be obtained, and the calibration coefficient corresponding to the depth value to be calibrated can be calculated later according to the calibration coefficient expression.
[0114] Step S105: performing binocular depth calibration according to the calibration coefficients.
[0115] In this embodiment, the depth value to be calibrated is substituted as the depth value before calibration Z into α=aZ2 +bZ+c is calculated to obtain the calibration coefficient corresponding to the depth value to be calibrated, and Z′=αZ is used to calculate the calibrated depth value Z′ corresponding to the depth value to be calibrated, thereby realizing the use of high-confidence trusted point cloud to calibrate the depth information of binocular perception and improve the perception accuracy of the binocular system.
[0116] See also Figure 5 In a specific embodiment, step S105 includes steps S1051 to S1054, and each step is described in detail below.
[0117] Step S1051, calculating the calibration error according to the calibration coefficient.
[0118] In this embodiment, after the calibrated depth value n corresponding to the pre-calibrated depth value m is calculated according to the calibration coefficient, the real depth value k of the object corresponding to the pre-calibrated depth value m is obtained.
[0119] Furthermore, the absolute value of the difference between the true depth value k and the calibrated depth value n is calculated to obtain the calibration error.
[0120] Step S1052: determining whether the calibration coefficient meets a condition according to a preset error threshold and the calibration error.
[0121] In this embodiment, it is determined whether the calibration accuracy of the current calibration coefficient meets the condition according to the preset error threshold and the calibration error, that is, it is determined whether the parameter value to be solved meets the condition.
[0122] Step S1053: If the calibration coefficient meets the conditions, perform binocular depth calibration according to the calibration coefficient.
[0123] In this embodiment, if the calibration coefficient meets the conditions, it can be known that the parameter value to be solved at this time can make the calibration coefficient calibrate the depth value more accurately, so the calibration coefficient can be used to perform binocular depth calibration.
[0124] Step S1054: if the calibration coefficient does not meet the condition, re-execute the step of acquiring the coordinates of multiple sampling points according to the input image.
[0125] In this embodiment, if the calibration coefficient does not meet the conditions, it can be known that the parameter value to be solved at this time cannot enable the calibration coefficient to calibrate the depth value more accurately. Therefore, step S102 needs to be re-executed to supplement the sampling points so that the fitting algorithm can obtain more reliable results, thereby calculating a more accurate parameter value to be solved.
[0126] In a specific embodiment, step S1051 includes: determining whether the calibration error is less than a preset error threshold; if the calibration error is less than the preset error threshold, the calibration coefficient meets the condition; if the calibration error is greater than or equal to the preset error threshold, the calibration coefficient does not meet the condition.
[0127] In this embodiment, it is determined whether the calibration error is less than a preset error threshold; if the calibration error is less than the preset error threshold, it can be known that the gap between the true depth value k and the calibrated depth value n is small, and the calibration coefficient meets the condition; if the calibration error is greater than or equal to the preset error threshold, it can be known that the gap between the true depth value k and the calibrated depth value n is large, and the calibration coefficient does not meet the condition.
[0128] The binocular depth calibration method proposed in this embodiment obtains a first depth map according to the disparity map corresponding to the input image, obtains a second depth map according to the credible point cloud corresponding to the input image; obtains a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively; constructs a calibration function based on the proportional relationship between the disparity error, the camera internal parameter error and the preset depth value; calculates the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient; and performs binocular depth calibration according to the calibration coefficient. In this way, a calibration function is constructed according to the relationship between the depth value and the disparity error and the camera internal parameter error respectively, and the calibration function is further solved using a credible point cloud with high confidence, and binocular depth calibration is performed according to the calibration coefficient obtained by the solution to improve the calibration accuracy.
[0129] Example 2
[0130] In addition, the present disclosure provides a binocular depth calibration device 600, see Figure 6 ,include:
[0131] An acquisition module 601 is used to acquire a first depth map according to a disparity map corresponding to an input image, and acquire a second depth map according to a trusted point cloud corresponding to the input image;
[0132] A sampling module 602 is used to obtain a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively;
[0133] A construction module 603 is used to construct a calibration function based on a proportional relationship between a parallax error, an intrinsic camera parameter error and a preset depth value;
[0134] A solution module 604, configured to calculate the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient;
[0135] The calibration module 605 is used to perform binocular depth calibration according to the calibration coefficient.
[0136] Optionally, a construction module 603 is used to obtain a conversion relationship between a preset depth value, a preset disparity value and a preset camera intrinsic parameter, the conversion relationship comprising: a quotient of the preset camera intrinsic parameter and the disparity value is the preset depth value; a first proportional relationship between the preset depth value and the disparity error is obtained according to the conversion relationship; a second proportional relationship between the preset depth value and the camera intrinsic parameter error is obtained according to the conversion relationship; and the calibration function is determined according to the first proportional relationship and the second proportional relationship.
[0137] Optionally, the solution module 604 is used to determine a first matrix according to the first sampling depth value, determine a sample vector according to each of the second sampling depth values, and determine a vector to be solved according to the parameter to be solved in the calibration function; based on the least squares method, solve the vector to be solved according to the first matrix and the sample vector to obtain the parameter value to be solved; and obtain the calibration coefficient according to the parameter value to be solved.
[0138] Optionally, the solution module 604 is used to determine the first product of the first matrix and the sample vector; determine the difference between the vector to be solved and the first product to obtain an error vector; and determine the sum of squares of the error vectors; differentiate the sum of squares with respect to the vector to be solved, and set the derivative equal to a target value to obtain a derivative equation; solve the derivative equation to obtain the parameter value to be solved.
[0139] Optionally, the calibration module 605 is used to calculate the calibration error based on the calibration coefficient; determine whether the calibration coefficient meets the conditions based on a preset error threshold and the calibration error; if the calibration coefficient meets the conditions, perform binocular depth calibration based on the calibration coefficient; if the calibration coefficient does not meet the conditions, re-execute the step of obtaining the coordinates of multiple sampling points based on the input image.
[0140] Optionally, the calibration module 605 is used to determine whether the calibration error is less than the preset error threshold; if the calibration error is less than the preset error threshold, the calibration coefficient meets the condition; if the calibration error is greater than or equal to the preset error threshold, the calibration coefficient does not meet the condition.
[0141] Optionally, the sampling module 602 is used to obtain multiple sampling point coordinates according to the input image; obtain a first sampling depth value corresponding to each sampling point coordinate from the first depth map; and obtain a second sampling depth value corresponding to each sampling point coordinate from the second depth map.
[0142] Optionally, the sampling module 602 is configured to perform instance segmentation on the input image to obtain a plurality of segmentation masks; and extract the coordinates of each sampling point in each segmentation mask.
[0143] The device provided in the embodiment of the present disclosure can execute the steps of the binocular depth calibration method provided in Example 1, which will not be described again to avoid repetition.
[0144] The binocular depth calibration device proposed in this embodiment obtains a first depth map according to the disparity map corresponding to the input image, and obtains a second depth map according to the credible point cloud corresponding to the input image; obtains multiple first sampling depth values and multiple second sampling depth values from the first depth map and the second depth map respectively; constructs a calibration function based on the proportional relationship between the disparity error, the camera internal parameter error and the preset depth value; calculates the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient; and performs binocular depth calibration according to the calibration coefficient. In this way, a calibration function is constructed according to the relationship between the depth value and the disparity error and the camera internal parameter error respectively, and the calibration function is further solved using a credible point cloud with high confidence, and binocular depth calibration is performed according to the calibration coefficient obtained by the solution to improve the calibration accuracy.
[0145] Example 3
[0146] In addition, an embodiment of the present disclosure provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the binocular depth calibration method described in Example 1 is implemented.
[0147] The device provided in the embodiment of the present disclosure can execute the steps of the binocular depth calibration method provided in Example 1, which will not be described again to avoid repetition.
[0148] Example 4
[0149] The embodiment of the present disclosure provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the binocular depth calibration method described in Embodiment 1 is implemented.
[0150] In this embodiment, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0151] The computer-readable storage medium provided in this embodiment can implement the binocular depth calibration method provided in Example 1. To avoid repetition, it will not be described here.
[0152] In all examples shown and described herein, any specific values should be interpreted as merely exemplary and not as limiting, and thus other examples of the exemplary embodiments may have different values.
[0153] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.
[0154] The above-mentioned embodiments only express several implementation methods of the present invention, and the description thereof is relatively specific and detailed, but it cannot be understood as limiting the scope of the present invention. It should be pointed out that for ordinary technicians in this field, several modifications and improvements can be made without departing from the concept of the present invention, which all belong to the protection scope of the present invention.
Claims
1. A binocular depth calibration method, characterized in that: include: Acquire a first depth map according to a disparity map corresponding to an input image, and acquire a second depth map according to a credible point cloud corresponding to the input image; Acquire a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively; Constructing a calibration function based on the proportional relationship between the parallax error, the camera intrinsic parameter error and the preset depth value; Calculating the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient; Perform binocular depth calibration according to the calibration coefficient.
2. The binocular depth calibration method according to claim 1, characterized in that: The method of constructing a calibration function based on a proportional relationship between a parallax error, an intrinsic camera parameter error and a preset depth value comprises: Acquire a conversion relationship between the preset depth value, the preset disparity value and a preset camera intrinsic parameter, wherein the conversion relationship includes: a quotient of the preset camera intrinsic parameter and the disparity value is the preset depth value; Acquire a first proportional relationship between the preset depth value and the parallax error according to the conversion relationship; Acquire a second proportional relationship between the preset depth value and the camera intrinsic parameter error according to the conversion relationship; The calibration function is determined according to the first proportional relationship and the second proportional relationship.
3. The binocular depth calibration method according to claim 1, characterized in that: The step of calculating the calibration function according to each of the first sampling depth values and each of the second sampling depth values to obtain a calibration coefficient includes: Determine a first matrix according to the first sampling depth value, determine a sample vector according to each of the second sampling depth values, and determine a vector to be solved according to the parameter to be solved in the calibration function; Solving the vector to be solved according to the first matrix and the sample vector based on the least squares method to obtain a parameter value to be solved; The calibration coefficient is obtained according to the parameter value to be solved.
4. The binocular depth calibration method according to claim 3, characterized in that: The method of solving the vector to be solved based on the first matrix and the sample vector based on the least squares method to obtain the parameter value to be solved includes: determining a first product of the first matrix and the sample vector; Determine the difference between the vector to be solved and the first product to obtain an error vector; and determine the sum of squares of the error vectors; Derivative the square sum with respect to the vector to be solved, and set the derivative equal to a target value to obtain a derivative equation; The derivative equation is solved to obtain the parameter value to be solved.
5. The binocular depth calibration method according to claim 1, characterized in that: The performing binocular depth calibration according to the calibration coefficient comprises: Calculating a calibration error based on the calibration coefficient; Determining whether the calibration coefficient meets a condition according to a preset error threshold and the calibration error; If the calibration coefficient meets the conditions, binocular depth calibration is performed according to the calibration coefficient; If the calibration coefficient does not meet the condition, the step of acquiring the coordinates of a plurality of sampling points according to the input image is performed again.
6. The binocular depth calibration method according to claim 5, characterized in that: The step of judging whether the calibration coefficient meets a condition according to a preset error threshold and the calibration error includes: Determining whether the calibration error is less than the preset error threshold; If the calibration error is less than the preset error threshold, the calibration coefficient satisfies the condition; If the calibration error is greater than or equal to the preset error threshold, the calibration coefficient does not meet the condition.
7. The binocular depth calibration method according to claim 1, characterized in that: The acquiring a plurality of first sampling depth values and a plurality of second sampling depth values from the first depth map and the second depth map respectively comprises: Acquire coordinates of multiple sampling points according to the input image; Acquire a first sampling depth value corresponding to the coordinates of each sampling point from the first depth map; The second sampling depth value corresponding to each of the sampling point coordinates is obtained from the second depth map.
8. The binocular depth calibration method according to claim 7, characterized in that: The step of acquiring coordinates of a plurality of sampling points according to the input image comprises: Performing instance segmentation on the input image to obtain multiple segmentation masks; The coordinates of each sampling point are extracted from each segmentation mask.
9. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the binocular depth calibration method according to any one of claims 1 to 7 is implemented.
10. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the binocular depth calibration method as described in any one of claims 1 to 7.