A monocular-binaric point cloud interpolation method, system and medium

Abstract

The application discloses a kind of single binocular point cloud interpolation method, system and medium, the method includes the following steps: based on camera system, first fiducial, camera reference plane and first phase and height mapping relationship formula executes single target operation, obtains to be quoted phase and height mapping relationship formula and rotation translation matrix;Based on camera system, binocular point cloud formula group and double target method are executed binocular point cloud calculation operation, obtain double target information, first binocular point cloud and hollow phase;Based on camera reference plane, to be quoted phase and height mapping relationship formula, rotation translation matrix, double target information, first binocular point cloud, hollow phase and point cloud interpolation calculation formula group executes point cloud interpolation operation;The application can be for the point cloud that missing in binocular three-dimensional measurement is high-precision interpolation, makes the reconstruction result of point cloud more complete, finally carries out point cloud smoothing processing, generates high-quality point cloud.

Description

Technical Field

[0001] This invention relates to the field of three-dimensional measurement technology, and in particular to a single- and binocular point cloud interpolation method, system, and medium. Background Technology

[0002] Currently, 3D measurement technology has been widely applied in various fields such as object grasping, workpiece quality inspection, and cultural relic digitization. The most commonly used 3D detection technology is phase profile measurement technology based on monocular or binocular systems; however, phase profile measurement technology based on binocular systems is prone to point cloud loss.

[0003] The prior art discloses a binocular measurement missing point cloud interpolation method based on phase pseudo-mapping (application number CN201910640592.6), which calculates the missing unknown point cloud based on the mapping relationship between pixels, but the accuracy of the point cloud obtained is low.

[0004] Prior art 2 discloses a three-dimensional measurement method combining monocular and binocular vision systems (published in the Acta Optica Sinica, 2008, Vol. 28, No. 7). In this method, the projector is equated to a camera when calibrating the system. Therefore, this method is not applicable to MEMS optomechanics and has certain limitations.

[0005] In summary, the present invention aims to solve the problems existing in the prior art 1 and prior art 2, and proposes a point cloud interpolation method that can improve the accuracy of point cloud interpolation, thereby improving the integrity of three-dimensional measurement, and has a wide range of applications. Summary of the Invention

[0006] The main objective of this invention is to propose a point cloud interpolation method that can improve the accuracy of point cloud interpolation, thereby improving the completeness of three-dimensional measurement, and has a wide range of applications.

[0007] To achieve the above objectives, one technical solution adopted by the present invention is to provide a single / binocular point cloud interpolation method, comprising the following steps:

[0008] Monocular system calibration steps:

[0009] Configure the camera system and the first reference object; set the camera reference plane and the first phase-to-height mapping formula; perform a single-target calibration operation based on the camera system, the first reference object, the camera reference plane and the first phase-to-height mapping formula to obtain the phase-to-height mapping formula to be referenced and the rotation and translation matrix;

[0010] Binocular point cloud computing steps:

[0011] Configure a binocular positioning method; set a binocular point cloud formula group; perform binocular point cloud computing operation based on the camera system, the binocular positioning method and the binocular point cloud formula group to obtain binocular positioning information, a first binocular point cloud and phase at the hole;

[0012] Binocular point cloud interpolation steps:

[0013] Set a set of point cloud interpolation calculation formulas; perform point cloud interpolation operations based on the camera reference plane, the phase-to-height mapping relationship to be referenced, the rotation and translation matrix, the binocular positioning information, the first binocular point cloud, the phase at the hole, and the set of point cloud interpolation calculation formulas.

[0014] As an improved solution, the camera system includes a monocular system and a binocular system; the binocular system is based on the monocular system; the camera reference plane includes: a first reference plane, a second reference plane, a third reference plane, and a fourth reference plane;

[0015] The binocular calibration method is the Zhang Zhengyou calibration method; the binocular point cloud formula group is configured with: the first x-value calculation formula, the first y-value calculation formula, and the first z-value calculation formula;

[0016] The point cloud interpolation calculation formula group includes: monocular point cloud computing formula and point cloud interpolation coordinate calculation formula.

[0017] As an improved approach, the single-target positioning operation includes:

[0018] The first reference plane is set as the camera reference plane. Based on the second reference plane, the third reference plane, the fourth reference plane and the camera reference plane, a difference calculation step is performed to obtain the first difference information, the second difference information and the third difference information.

[0019] Substituting the first difference information into the first phase-height mapping formula yields the first height formula; substituting the second difference information into the first phase-height mapping formula yields the second height formula; substituting the third difference information into the first phase-height mapping formula yields the third height formula.

[0020] Identify the first constant in the first phase-to-height mapping formula, and calculate the constant value corresponding to the first constant by combining the first height formula, the second height formula, and the third height formula; update the first constant in the first phase-to-height mapping formula based on the constant value to obtain the phase-to-height mapping formula to be referenced.

[0021] A first pixel of the monocular system is defined; a first reference phase of the first pixel relative to the first reference object is obtained based on the monocular system; a second reference phase of the first pixel relative to the first reference plane is obtained based on the monocular system; a third reference phase of the first pixel relative to the second reference plane is obtained based on the monocular system; a reference plane determination step is performed based on the first reference phase, the second reference phase, and the third reference phase to obtain a first reference plane to be calibrated;

[0022] Based on the phase-height mapping relationship to be referenced and the first calibration reference surface, the monocular system is calibrated using the phase-height mapping method to obtain the rotation and translation matrix.

[0023] As an improved approach, the difference calculation step includes:

[0024] Define a second pixel in the monocular system; obtain a first planar phase of the second pixel with respect to the camera reference plane based on the monocular system; obtain a second planar phase of the second pixel with respect to the second reference plane based on the monocular system; obtain a third planar phase of the second pixel with respect to the third reference plane based on the monocular system; obtain a fourth planar phase of the second pixel with respect to the fourth reference plane based on the monocular system.

[0025] Calculate the first phase difference between the second plane phase and the first plane phase; calculate the second phase difference between the third plane phase and the first plane phase; calculate the third phase difference between the fourth plane phase and the first plane phase;

[0026] The monocular system is used to obtain a first height value of the first reference plane; the monocular system is used to obtain a second height value of the second reference plane; the monocular system is used to obtain a third height value of the third reference plane; and the monocular system is used to obtain a fourth height value of the fourth reference plane.

[0027] Calculate the first height difference between the second height value and the first height value; calculate the second height difference between the third height value and the first height value; calculate the third height difference between the fourth height value and the first height value;

[0028] Integrate the first phase difference value and the first height difference value to obtain the first difference value information; integrate the second phase difference value and the second height difference value to obtain the second difference value information; integrate the third phase difference value and the third height difference value to obtain the third difference value information;

[0029] The steps for determining the reference plane include:

[0030] Calculate the first absolute value of the difference between the first reference phase and the second reference phase; calculate the second absolute value of the difference between the first reference phase and the third reference phase; compare the first absolute value and the second absolute value to determine whether the first absolute value is greater than the second absolute value; if yes, select the first reference plane as the first calibration reference plane; if no, select the second reference plane as the first calibration reference plane.

[0031] As an improved solution, the binocular point cloud computing operation includes:

[0032] The binocular system is calibrated using the Zhang Zhengyou calibration method to obtain a first focal length parameter, a first baseline parameter, pixel center information, and pixel size information; the first focal length parameter, the first baseline parameter, the pixel center information, and the pixel size information are integrated to obtain the binocular calibration information;

[0033] In the binocular system, a first camera and a second camera are selected; based on the first camera, a plurality of first phases to be screened of the first reference object are acquired; based on the second camera, a plurality of second phases to be screened of the first reference object are acquired; based on the plurality of first phases to be screened and the plurality of second phases to be screened, a corresponding point matching step is performed;

[0034] The corresponding point matching step includes: selecting a first corresponding phase from a plurality of first phases to be screened; selecting a second corresponding phase from a plurality of second phases to be screened that matches the phase value of the first corresponding phase; identifying the first phase coordinates corresponding to the first corresponding phase and the second phase coordinates corresponding to the second corresponding phase; identifying the first row value and the first column value in the first phase coordinates; identifying the second row value in the second phase coordinates; calculating the difference between the second row value and the first row value to obtain a first corresponding disparity value;

[0035] After performing the corresponding point matching step, the first row value and the first focal length parameter are substituted into the first x-value calculation formula to obtain the second x-value calculation formula; the first column value and the first focal length parameter are substituted into the first y-value calculation formula to obtain the second y-value calculation formula; the first focal length parameter, the first baseline parameter, and the first corresponding disparity value are substituted into the first z-value calculation formula to obtain the z-coordinate value; the z-coordinate value is then substituted into the second x-value calculation formula and the second y-value calculation formula to obtain the x-coordinate value and the y-coordinate value, respectively.

[0036] Generate stereo point cloud coordinates based on the x-coordinate, y-coordinate, and z-coordinate values; construct the first stereo point cloud based on the stereo point cloud coordinates.

[0037] As an improved solution, the binocular point cloud computing operation further includes:

[0038] When performing the corresponding point matching step, it is determined whether there is a second corresponding phase among the several second phases to be screened that matches the phase value of the first corresponding phase. If there is no second corresponding phase, the first corresponding phase is set as the phase at the hole.

[0039] As an improved solution, the point cloud interpolation operation includes:

[0040] The first camera or the second camera corresponding to the phase at the hole is set as the reference camera; the first hole pixel corresponding to the phase at the hole in the reference camera is identified; the first hole judgment phase of the first hole pixel relative to the first reference plane is obtained based on the reference camera; the second hole judgment phase of the first hole pixel relative to the second reference plane is obtained based on the reference camera; the reference plane determination step is performed based on the hole phase, the first hole judgment phase, and the second hole judgment phase to obtain the first interpolation calculation reference plane;

[0041] Set the first hole judgment phase or the second hole judgment phase corresponding to the first interpolation calculation reference plane as the first phase to be calculated; calculate the difference between the phase at the hole and the first phase to be calculated to obtain the phase variable at the hole; substitute the phase variable at the hole into the mapping relationship between the phase to be referenced and the height to obtain the height difference at the hole; set the height difference at the hole as the z-coordinate value of the monocular point cloud;

[0042] Based on the dual-target positioning information, an image coordinate calculation formula is set; based on the dual-target positioning information, a pixel coordinate system corresponding to the reference camera is set; the hole pixel coordinates of the first hole in the pixel coordinate system are obtained; the hole pixel coordinates are substituted into the image coordinate calculation formula to obtain the hole image coordinates; the hole image coordinates are substituted into the monocular point cloud computing formula to obtain the monocular point cloud coordinate relationship to be referenced.

[0043] Substituting the monocular point cloud coordinate relationship to be referenced and the rotation and translation matrix into the point cloud interpolation coordinate calculation formula, we obtain the monocular point cloud x-coordinate value and the monocular point cloud y-coordinate value; substituting the monocular point cloud z-coordinate value, the monocular point cloud x-coordinate value and the monocular point cloud y-coordinate value into the point cloud interpolation coordinate calculation formula, we obtain the point cloud interpolation coordinates.

[0044] In the first binocular point cloud, the binocular point cloud coordinates corresponding to the phase at the hole are replaced with the point cloud interpolation coordinates to obtain a complete binocular point cloud.

[0045] As an improved solution, the single- and binocular point cloud interpolation method further includes: performing point cloud smoothing processing based on the first z-value calculation formula, the phase at the hole, and the complete binocular point cloud;

[0046] The point cloud smoothing process includes:

[0047] Based on the phase at the hole, a connected region is confirmed in the complete binocular point cloud; a fusion template is configured; based on the fusion template, an expanded adjacent region corresponding to the connected region is set; a first interpolation coordinate and a second interpolation coordinate are identified in the expanded adjacent region; a first depth value corresponding to the first interpolation coordinate is identified; and a second depth value corresponding to the second interpolation coordinate is identified.

[0048] Set a deviation threshold, calculate the absolute value of the deviation between the first depth value and the second depth value; determine whether the absolute value of the deviation is less than the deviation threshold; if it is less, substitute the first depth value and the second depth value into the first z-value calculation formula respectively to obtain the first smoothed disparity value and the second smoothed disparity value.

[0049] Calculate the average disparity value of the first smoothed disparity value and the second smoothed disparity value; replace the first depth value in the first interpolation coordinates with the average disparity value; replace the second depth value in the second interpolation coordinates with the average disparity value.

[0050] The present invention also provides a monocular and binocular point cloud interpolation system, comprising:

[0051] Single-target positioning module, binocular point cloud computing module, and point cloud interpolation module;

[0052] The single-target calibration module is used to configure the camera system and the first reference object; the single-target calibration module is also used to set the camera reference plane and the first phase-to-height mapping formula; the single-target calibration module performs a single-target calibration operation based on the camera system, the first reference object, the camera reference plane and the first phase-to-height mapping formula to obtain the phase-to-height mapping formula to be referenced and the rotation and translation matrix;

[0053] The binocular point cloud computing module is used to configure the binocular positioning method and set the binocular point cloud formula group; the binocular point cloud computing module performs binocular point cloud computing operation based on the camera system, the binocular positioning method and the binocular point cloud formula group to obtain binocular positioning information, the first binocular point cloud and the phase at the hole;

[0054] The point cloud interpolation module is used to set a set of point cloud interpolation calculation formulas; the point cloud interpolation module performs point cloud interpolation operations based on the camera reference plane, the phase-to-height mapping relationship to be referenced, the rotation and translation matrix, the binocular positioning information, the first binocular point cloud, the phase at the hole, and the set of point cloud interpolation calculation formulas.

[0055] The present invention also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of the mono- and binocular point cloud interpolation method.

[0056] The beneficial effects of this invention are:

[0057] 1. The monocular and binocular point cloud interpolation method described in this invention can achieve high-precision interpolation of missing point clouds in binocular 3D measurement, making the point cloud reconstruction results more complete. At the same time, this method adopts a monocular and binocular point cloud fusion architecture design for point cloud interpolation, and finally performs point cloud smoothing processing to generate high-quality point clouds. This method is applicable to different cameras or optical engines, has low limitations, strong universality, makes up for the shortcomings of existing technologies, and has extremely high application value and forward-looking.

[0058] 2. The monocular and binocular point cloud interpolation system described in this invention can achieve high-precision interpolation of missing point clouds in binocular 3D measurement through the cooperation of a monocular positioning module, a binocular point cloud computing module, and a point cloud interpolation module, making the reconstructed point cloud results more complete. Simultaneously, this system adopts a monocular and binocular point cloud fusion architecture for point cloud interpolation and finally performs point cloud smoothing processing to generate high-quality point clouds. This system is applicable to different cameras or optical engines, has low limitations, strong universality, and overcomes the shortcomings of existing technologies, possessing extremely high application value and forward-looking potential.

[0059] 3. The computer-readable storage medium described in this invention can guide the cooperation of a single-target positioning module, a binocular point cloud computing module, and a point cloud interpolation module, thereby achieving high-precision interpolation of the missing point cloud in binocular 3D measurement, making the point cloud reconstruction result more complete, and finally performing point cloud smoothing processing to generate a high-quality point cloud. It has low limitations, strong universality, makes up for the shortcomings of the prior art, and effectively improves the operability of the single and binocular point cloud interpolation method. Attached Figure Description

[0060] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0061] Figure 1 This is a flowchart of the single / binocular point cloud interpolation method described in Embodiment 1 of the present invention;

[0062] Figure 2 This is a schematic diagram of the specific process of the single and binocular point cloud interpolation method described in Embodiment 1 of the present invention;

[0063] Figure 3 This is a schematic diagram of the reference plane as described in Embodiment 1 of the present invention;

[0064] Figure 4 This is a schematic diagram of cavity confirmation as described in Embodiment 1 of the present invention;

[0065] Figure 5 This is a schematic diagram of the point cloud before interpolation as described in Embodiment 1 of the present invention;

[0066] Figure 6 This is a schematic diagram of point cloud interpolation as described in Embodiment 1 of the present invention;

[0067] Figure 7 This is a schematic diagram of the fusion template described in Embodiment 1 of the present invention;

[0068] Figure 8 This is an architecture diagram of the single / binocular point cloud interpolation system described in Embodiment 2 of the present invention. Detailed Implementation

[0069] The preferred embodiments of the present invention will now be described in detail with reference to the accompanying drawings, so that the advantages and features of the present invention can be more easily understood by those skilled in the art, thereby providing a clearer and more explicit definition of the scope of protection of the present invention.

[0070] In the description of this invention, it should be noted that the embodiments described in this invention are only some embodiments of this invention, not all embodiments; based on the embodiments of this invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this invention.

[0071] In the description of this invention, it should be noted that the terms "first", "second", "third" and "fourth" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0072] In the description of this invention, it should be noted that, unless otherwise explicitly specified and limited, the terms "phase and height mapping relationship", "binocular point cloud formula set", "binocular calibration information", "phase at the hole", "point cloud interpolation calculation formula set", "x-value calculation formula", "y-value calculation formula", "z-value calculation formula", "monocular point cloud computing formula", "point cloud interpolation coordinate calculation formula", "reference surface to be calibrated", "planar phase", "pixel center information", "pixel size information", "phase to be screened", "same-name phase", "same-name disparity value", "reference camera", "pixel at the hole", "hole judgment phase", "interpolation calculation reference surface", "phase to be calculated", "phase variable at the hole", "height difference at the hole", "image coordinate calculation formula", "hole pixel coordinates", "image coordinates at the hole", "point cloud interpolation coordinates", "complete binocular point cloud", "connected region", "fusion template", "dilated adjacent region", "interpolation coordinates", "depth value", "deviation threshold", "absolute deviation value", "smoothed disparity value", "average disparity value", "monocular calibration module", "binocular point cloud computing module", and "point cloud interpolation module" should be interpreted broadly. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0073] In the description of this invention, it should be noted that MEMS (Micro-Electro-Mechanical System) is a micro-electro-mechanical system.

[0074] Example 1

[0075] This embodiment provides a method for single and binocular point cloud interpolation, such as... Figures 1-7 As shown, it includes the following steps:

[0076] In this embodiment, the left and right cameras are each treated as two separate monocular systems. When single-target timing is required, both monocular systems are calibrated individually. When dual-target timing is required, the binocular system formed by the two monocular systems is calibrated. Correspondingly, in this embodiment, the left camera is used as a separate example. During implementation, point cloud interpolation operations are performed simultaneously on the left and right cameras based on this method. The specific operation steps are as follows:

[0077] S100, Monocular system calibration steps, specifically including:

[0078] S110. Configure the camera system and the first reference object; set the camera reference plane and the first phase-to-height mapping formula; perform a single-target calibration operation based on the camera system, the first reference object, the camera reference plane and the first phase-to-height mapping formula to obtain the phase-to-height mapping formula to be referenced and the rotation and translation matrix;

[0079] Specifically, the camera system includes a monocular system and a binocular system; the binocular system is based on the monocular system; the monocular system is based on a left camera or a right camera, and the binocular system is based on a left camera and a right camera. In this embodiment, the optical engine adopts a MEMS optical engine; the camera reference plane includes: a first reference plane, a second reference plane, a third reference plane, and a fourth reference plane.

[0080] Specifically, in step S110, the following steps are performed, and the main purpose of step S110 is to calibrate the monocular system.

[0081] Specifically, the initial size of the image pixels acquired by the camera is set to n*m, which corresponds to the generation of a camera pixel coordinate system. However, because the height of the object varies and the distance between the object and the camera differs, for a given pixel of the camera, each time it acquires a certain phase of the measured object... They are also different; therefore, in this embodiment, the phase-height mapping method is used to calibrate the monocular system. The principle of the specific calibration method is based on the paper "Large-scale three-dimensional object measurement: a practical coordinatemapping and image data-patching method". The necessary condition for using this method is the confirmation of the camera's reference plane and the confirmation of the specific mapping relationship between the object's phase and height. Before performing the specific operation, the following analysis is performed: A first reference plane is set for the camera being measured (left or right camera); the height difference between the measured object (first reference object) and this first reference plane is set to... The camera captures a certain phase on that plane. Then, based on the steps described above, the camera acquires a certain phase of the object being measured. What can be obtained is the phase difference between the measured object and the first reference plane. Based on relevant knowledge in the field of optical measurement, it can be concluded that... and The relationship satisfies formula (1), that is:

[0082] (1)

[0083] Specifically, in this embodiment, formula (1) corresponds to the first phase-height mapping relationship, based on the calculations between pixels. , where h The height of the object being measured. This is the height of the first reference plane; therefore , , Once these three constants are determined, the corresponding... Thus, the height of the object is obtained; therefore, the following steps are performed to calculate the constant through the monocular positioning operation of the monocular system, and then obtain the corresponding phase-height mapping relationship, i.e., formula (1).

[0084] Specifically, the single-target calibration operation includes: firstly, setting four mutually parallel reference planes (i.e., camera reference planes), and selecting a corresponding reference plane from these four mutually parallel reference planes, i.e., setting the first reference plane as the camera reference plane; in this embodiment, the camera reference planes are arranged in the following order: first reference plane, third reference plane, fourth reference plane, and second reference plane. Calculations are performed using the first reference plane as the camera reference plane, and then a MEMS optical engine is used to project the reference planes; subsequently, a difference calculation step is performed based on the second reference plane, the third reference plane, the fourth reference plane, and the camera reference plane to obtain first difference information, second difference information, and third difference information;

[0085] Specifically, the difference calculation steps include: setting a second pixel in the monocular system; obtaining a first planar phase of the second pixel with respect to the camera reference plane based on the monocular system; obtaining a second planar phase of the second pixel with respect to the second reference plane based on the monocular system; obtaining a third planar phase of the second pixel with respect to the third reference plane based on the monocular system; obtaining a fourth planar phase of the second pixel with respect to the fourth reference plane based on the monocular system; calculating a first phase difference between the second planar phase and the first planar phase; calculating a second phase difference between the third planar phase and the first planar phase; calculating a third phase difference between the fourth planar phase and the first planar phase; and obtaining a first phase difference between the second planar phase and the first planar phase based on the monocular system. A first height value for a first reference plane; a second height value for a second reference plane obtained based on the monocular system; a third height value for a third reference plane obtained based on the monocular system; a fourth height value for a fourth reference plane obtained based on the monocular system; a first height difference between the second height value and the first height value calculated; a second height difference between the third height value and the first height value calculated; a third height difference between the fourth height value and the first height value calculated; integrating the first phase difference value and the first height difference value to obtain first difference information; integrating the second phase difference value and the second height difference value to obtain second difference information; integrating the third phase difference value and the third height difference value to obtain third difference information.

[0086] Correspondingly, in this embodiment, the following example is used to explain the difference calculation step: obtaining the first plane phase of a certain pixel (i.e., the second pixel) of the camera in the first reference plane. And sequentially acquire the phase of a certain pixel of the camera (i.e., the second pixel) in the third reference plane, the fourth reference plane, and the third plane of the second reference plane. Phase of the fourth plane Second plane phase Correspondingly, obtain the height of the first plane in the camera relative to the first reference plane. (i.e., the first height value), and then sequentially obtain the height of the third plane in the camera relative to the third reference plane, the fourth reference plane, and the second reference plane. (i.e., the third height value), the fourth plane height Second plane height (i.e., the second altitude value); at this point, the corresponding difference is calculated, i.e., the phase of the third plane. Phase of the fourth plane Second plane phase Phase with the first plane respectively The difference; the height of the third plane Fourth plane height Second plane height Height of the first plane respectively The difference; correspondingly, taking each plane as a distinction, the above differences are organized to obtain the difference information for each reference plane; that is, the difference information for the third reference plane (i.e., the second difference information) is: The difference information corresponding to the fourth reference plane (i.e., the third difference information) is: The difference information corresponding to the second reference plane (i.e., the first difference information) is: In this embodiment, the corresponding reference plane schematic diagram is as follows: Figure 3 As shown;

[0087] Specifically, after obtaining the first difference information, the second difference information, and the third difference information, the first difference information is substituted into the first phase-height mapping relationship to obtain the first height relationship; the second difference information is substituted into the first phase-height mapping relationship to obtain the second height relationship; the third difference information is substituted into the first phase-height mapping relationship to obtain the third height relationship, that is: according to the above difference information, they are substituted into formula (1) respectively to obtain the mapping relationship corresponding to the third reference plane, the fourth reference plane, and the second reference plane respectively, that is, formula (2), formula (3), and formula (4), formula (2) is the second height relationship, formula (3) is the third height relationship, and formula (4) is the first height relationship, as follows:

[0088] , (2)

[0089] , (3)

[0090] , (4)

[0091] Correspondingly, after obtaining the relation, the first constant in the first phase-to-height mapping relation is identified, and the constant value corresponding to the first constant is calculated by combining the first height relation, the second height relation, and the third height relation; based on the constant value, the first constant in the first phase-to-height mapping relation is updated to obtain the phase-to-height mapping relation to be referenced, that is: the three constants in formula (1). , and All of these are the first constants. By solving the three relationships simultaneously, the three constants in formula (1) can be obtained. , and Correspondingly, the obtained , and The specific values ​​(i.e. constant values) are replaced in formula (1) to obtain the specific phase-height mapping relationship of the measured object that can be referenced (i.e. the phase-height mapping relationship to be referenced).

[0092] Correspondingly, after determining the phase and height mapping relationship, it is necessary to identify the camera's reference plane. Specifically, the first pixel of the monocular system is set; a first reference phase of the first pixel relative to the first reference object is obtained based on the monocular system; a second reference phase of the first pixel relative to the first reference plane is obtained based on the monocular system; a third reference phase of the first pixel relative to the second reference plane is obtained based on the monocular system; a reference plane determination step is performed based on the first reference phase, the second reference phase, and the third reference phase to obtain a first reference plane to be calibrated; the monocular system is calibrated using a phase-height mapping method based on the phase-height mapping relationship to be referenced and the first reference plane to be calibrated to obtain the rotation and translation matrix.

[0093] Specifically, the reference plane determination step includes: calculating a first absolute value of the difference between the first reference phase and the second reference phase; calculating a second absolute value of the difference between the first reference phase and the third reference phase; comparing the first absolute value and the second absolute value to determine whether the first absolute value is greater than the second absolute value; if so, selecting the first reference plane as the first calibration reference plane; if not, selecting the second reference plane as the first calibration reference plane; in this embodiment, to ensure accuracy, the reference plane is selected between the first reference plane and the second reference plane; the reference plane determination step and the subsequent single-target calibration operation in this embodiment are as follows: referencing a phase of the measured object (first reference object) acquired by a certain pixel (i.e., the first pixel) of the camera. (i.e., the first reference phase), and then obtain the first judgment phase of a certain pixel of the camera on the first reference plane. (i.e., the second reference phase); acquiring the second judgment phase of a certain pixel (i.e., the first pixel) on the second reference plane. (i.e., the third reference phase); calculation and Calculate the first absolute value of the difference. and The second absolute value of the difference; comparing two absolute values, i.e. The first and second absolute values ​​are compared. If the first absolute value is greater than the second absolute value, the first reference plane is used as the reference plane for this camera (i.e., the first calibration reference plane). If the first absolute value is not greater than the second absolute value, the second reference plane is used as the reference plane for this camera (i.e., the first calibration reference plane). Correspondingly, after determining the reference plane, based on the reference plane (i.e., the first calibration reference plane), the phase-height mapping relationship obtained above (the phase-height mapping relationship to be referenced), and based on the phase-height mapping method (i.e., the phase-height mapping method), the left camera monocular system is calibrated, and finally the rotation and translation matrix M of the left camera is obtained. l Correspondingly, the single-target localization of the right camera follows the same principle as above, ultimately yielding the rotation and translation matrix M of the right camera. r The rotation and translation matrix is ​​used for subsequent point cloud coordinate calculation. In this embodiment, the rotation and translation matrix is ​​represented as follows:

[0094] .

[0095] S200, binocular point cloud computing steps, specifically include:

[0096] S210. Configure the binocular positioning method; set the binocular point cloud formula group; perform binocular point cloud computing operation based on the camera system, the binocular positioning method and the binocular point cloud formula group to obtain binocular positioning information, the first binocular point cloud and the phase at the hole;

[0097] Specifically, the binocular calibration method is the Zhang Zhengyou calibration method; the binocular point cloud formula group is configured with: a first x-value calculation formula, a first y-value calculation formula, and a first z-value calculation formula; correspondingly, the specific purpose of step S200 is to perform binocular point cloud calculation and determine the voids in the point cloud;

[0098] Specifically, the binocular point cloud computing operation includes: combining the left and right cameras into a binocular system; firstly, using the Zhang Zhengyou calibration method to perform binocular calibration on the binocular system, thereby obtaining the camera's focal length f (i.e., the first focal length parameter), baseline T (the first baseline parameter), pixel center information, and pixel size information; the pixel center information includes specific data of the pixel center, and the pixel size information includes the pixel row and column size and the pixel length and width values; integrating the first focal length parameter, the first baseline parameter, the pixel center information, and the pixel size information to obtain the binocular calibration information; selecting a first camera and a second camera in the binocular system; in this embodiment, the first camera is the left camera, and the second camera is the right camera;

[0099] Specifically, corresponding point matching is performed based on the phases of the first reference object captured by the left and right cameras respectively. That is, several first phases to be selected of the first reference object are obtained based on the first camera; several second phases to be selected of the first reference object are obtained based on the second camera; and corresponding point matching steps are performed based on the several first phases to be selected and the several second phases to be selected. The several first phases to be selected are the several phases obtained by the first camera for the first reference object; and the several second phases to be selected are the several phases obtained by the second camera for the first reference object.

[0100] Specifically, the corresponding point matching step includes: selecting a first corresponding phase from a plurality of first phases to be screened; selecting a second corresponding phase from a plurality of second phases to be screened that matches the phase value of the first corresponding phase; identifying the first row value and the first column value in the first corresponding phase; identifying the second row value in the second corresponding phase; calculating the difference between the second row value and the first row value to obtain the first corresponding disparity value, that is: when acquiring the phase of a certain pixel of the left camera... (i.e., the first phase of the same name), and acquire the phase in the right camera. New phase with the same phase value (i.e., the second corresponding phase); identify the first phase coordinates of the first corresponding phase and the second phase coordinates of the second corresponding phase; identify the first row value and the first column value in the first phase coordinates; identify the second row value in the second phase coordinates, calculate the difference between the second row value and the first row value, and obtain the first corresponding disparity value, that is: set the phase coordinates of these two phases in the two camera coordinate planes as (x l ,y l (i.e., first phase coordinates), (x) r ,y r (i.e., the second phase coordinate); the first row value is: x l The value of the second row is: x r The value of the first column is: y l Correspondingly, the disparity between two phases is calculated based on the phase coordinates, i.e., d=x. r -x l (i.e., the first parallax value); based on the knowledge of point cloud computing in binocular systems, the calculation formula group for binocular point clouds can be set as (including formulas 5, 6, and 7), where formula 5 is the calculation formula for the first x value, formula 6 is the calculation formula for the first y value, and formula 7 is the calculation formula for the first z value, as detailed below:

[0101] (5);

[0102] (6);

[0103] (7);

[0104] Correspondingly, where x r x l f, T, and d are all known values. Substituting them into the solution yields the binocular point cloud coordinates (x, y, z) corresponding to each phase. Specifically, after performing the corresponding point matching step, the first row value and the first focal length parameter are substituted into the first x-value calculation formula to obtain the second x-value calculation formula; the first column value and the first focal length parameter are substituted into the first y-value calculation formula to obtain the second y-value calculation formula; the first focal length parameter, the first baseline parameter, and the first corresponding disparity value are substituted into the first z-value calculation formula to obtain the z-coordinate value; the z-coordinate value is then substituted into the second x-value calculation formula and the second y-value calculation formula to obtain the x-coordinate value and the y-coordinate value; binocular point cloud coordinates (x, y, z) are generated based on the x-coordinate value, the y-coordinate value, and the z-coordinate value; and the binocular point cloud is constructed based on several binocular point cloud coordinates (x, y, z), thus obtaining the first binocular point cloud.

[0105] Specifically, after calculating the binocular point cloud, it is necessary to search for point cloud holes to determine them. Then, the binocular point cloud is interpolated using the monocular point cloud to complete the main interpolation process. Therefore, the binocular point cloud computing operation also includes: when performing the corresponding point matching step, determining whether there is a second corresponding phase among several second phases to be screened that matches the phase value of the first corresponding phase. If not, the first corresponding phase is set as the phase at the hole. That is, in this embodiment, the point cloud hole determination strategy is: if there is a phase of a certain pixel in the left camera... (i.e., the first phase with the same name in this step), and correspondingly, based on this phase The system acquires the same phase values ​​from the right camera and calculates the disparity. During disparity calculation, if there is no disparity d(i,j), it means that the phase captured by the right camera does not contain any values ​​that are identical to the phase values ​​captured by the right camera. The same phase (i.e., the second phase with the same name in this step) indicates that the phase The area at the right camera is empty; correspondingly, the same principle applies to the right camera area, meaning that there is a phase of a certain pixel in the right camera area. Regarding phase The parallax does not exist, meaning that there is no parallax in the phase captured by the left camera. The same phase indicates that the phase The point is a void; the phase corresponding to the void is the phase at the void, and the monocular point cloud is then calculated based on the phase at the void.

[0106] S300, binocular point cloud interpolation steps, specifically include:

[0107] S310. Set a point cloud interpolation calculation formula group; perform point cloud interpolation operation based on the camera reference plane, the phase-to-height mapping relationship to be referenced, the rotation and translation matrix, the binocular positioning information, the first binocular point cloud, the phase at the hole, and the point cloud interpolation calculation formula group; specifically, the point cloud interpolation calculation formula group is configured with: monocular point cloud computing formula and point cloud interpolation coordinate calculation formula;

[0108] Specifically, in this step, the interpolation of point clouds at holes is mainly performed. Therefore, for the phase at each hole, the following point cloud interpolation operation is performed respectively.

[0109] Specifically, the point cloud interpolation operation includes: firstly, confirming the phase at the hole corresponds to the pixel at the first hole on the camera (i.e., the first camera or the second camera); obtaining the first hole judgment phase of the pixel at the first hole on the first reference plane; obtaining the second hole judgment phase of the pixel at the first hole on the second reference plane; executing the reference plane selection strategy (i.e., the reference plane determination step) in the first step based on the hole phase, the first hole judgment phase, and the second hole judgment phase; that is, calculating the absolute value of the difference between the hole phase and the first hole judgment phase; calculating the absolute value of the difference between the hole phase and the second hole judgment phase; then selecting the plane corresponding to the larger absolute value as the reference plane (i.e., the first interpolation calculation reference plane); and then calculating the difference between the hole phase and the hole judgment phase (i.e., the first phase to be calculated) corresponding to the plane corresponding to the larger absolute value to obtain the corresponding hole. ; the cavity Substituting these values ​​into the specific, replaceable, and referable phase-height mapping formula (i.e., the phase-height mapping formula to be referenced), the hole can then be determined. (i.e., the height difference at the cavity); the cavity at this time That is, the corresponding monocular point cloud coordinates Z w (i.e., the z-coordinate value of the monocular point cloud); correspondingly, subsequent steps involve X... w and Y w Solve the problem;

[0110] First, based on the relationship between pixels and camera coordinates and the above binocular positioning information, formulas (8) and (9) are set as follows (formulas 8 and 9 are both image coordinate calculation formulas):

[0111] (8);

[0112] (9);

[0113] Correspondingly, The pixel center in the pixel center information. It is the ratio of focal length to pixel length. The ratio of focal length to pixel width is given by the fact that after camera calibration, the pixel center, pixel length, and pixel width can all be known from the calibration results, while the focal length is a known quantity. Then, the pixel coordinate system corresponding to the reference camera is set by the relevant values ​​of the pixel row and pixel column set at the beginning, and the pixel coordinates (i, j) of the pixel at the first hole in the pixel coordinate system (i, j) (i.e., hole pixel coordinates) are obtained. The pixel coordinates (i, j) are substituted into formulas (8) and (9) for simultaneous calculation, and the image coordinates (x1, y1) of the pixel at the first hole corresponding to the phase at the hole (i.e., hole image coordinates) can be obtained.

[0114] Correspondingly, based on relevant knowledge in the field of optical measurement, and the coordinate system with the camera as the origin and the coordinate system where the above image coordinates are located, the camera point cloud coordinate calculation formulas (10) and (11) can be set (Formulas 10 and 11 are both monocular point cloud computing formulas); correspondingly, Formulas (10) and (11) are as follows:

[0115] (10);

[0116] (11);

[0117] Correspondingly, substituting the above image coordinates (x1, y1) into formulas (10) and (11) yields formulas (10) and (11) that can be used for calculation (formulas 10 and 11 are the monocular point cloud coordinate relationships to be referenced); correspondingly, X c Y c and Z c The coordinates formed (X) c Y c Z c Let be the point cloud coordinates in the coordinate system with the camera as the origin; correspondingly, if point cloud interpolation is to be performed, the specific values ​​of the point cloud coordinates in the coordinate system with the camera as the origin must be calculated. The calculation here can be performed based on the relevant values ​​of the reference plane coordinate system; therefore, based on the relationship between the reference plane coordinate system and the coordinate system with the camera as the origin, the calculation formula (12) for the reference plane point cloud coordinates (i.e., the point cloud interpolation coordinate calculation formula) is set as follows:

[0118] (12)

[0119] Correspondingly, because M l Since the rotation and translation matrix obtained through calibration in the first step is a known quantity, M is... l (i.e., rotation and translation matrices) and (X) c Y c Z c Substituting the values ​​into formula (12), and combining them with formulas (10) and (11) (i.e., the monocular point cloud coordinate relationship), and then substituting them into formula (12) again, the corresponding X can be calculated. w (i.e., the x-coordinate value of the monocular point cloud) and y-coordinate value w (Monocular point cloud y-coordinate value); correspondingly, combined with the Z-coordinate value obtained at the beginning of this step. w The point cloud coordinates (X) under the reference plane are obtained. w Y w Z w Finally, (X) w Y wZ w Substituting this into formula (12), we get the corresponding (X) c Y c Z c The specific value (i.e., the point cloud interpolation coordinates) is (X). c Y c Z c The coordinates of the point cloud corresponding to the hole are shown below. In the first binocular point cloud, the binocular point cloud coordinates corresponding to the phase at the hole are replaced with the point cloud interpolation coordinates to obtain a complete binocular point cloud. At this point, point cloud interpolation is complete. Specifically, a schematic diagram before point cloud interpolation is shown below. Figure 5 As shown in the diagram, the point cloud after interpolation is as follows: Figure 6 As shown.

[0120] Specifically, in this embodiment, in order to achieve better interpolation effect and eliminate the depth misalignment phenomenon at the interpolation edge, relevant point cloud smoothing processing is performed to make the interpolation edge transition smoothly; therefore, point cloud smoothing processing operation is performed based on the first z-value calculation formula, the phase at the hole, and the complete binocular point cloud.

[0121] Specifically, the point cloud smoothing operation includes:

[0122] In the binocular point cloud, the coordinates of holes are marked as 0, and non-hole locations are marked as 1. Connectivity of each hole is determined in pixels. In this embodiment, to achieve the best interpolation effect, the maximum connected region is determined, thus obtaining the maximum connected region. Correspondingly, the maximum connected region is illustrated in the hole determination diagram of this embodiment. Figure 4 The central color difference portion;

[0123] First, a fusion template is used to dilate the maximum connected component from the third step. Correspondingly, the fusion template contains dilated pixels, each with a 3x3 structuring element, whose origin is the center pixel. In this embodiment, the fusion template is set based on the maximum connected component, including adjacent regions (i.e., dilated adjacent regions) and dilated regions. The dilated region is the area defined by the dilated pixels around the outer contour of the maximum connected component, and the adjacent region is the region adjacent to the outer contour of the maximum connected component. Correspondingly, the principle of depth misalignment at the interpolation edge is analyzed, revealing that edge misalignment is caused by the inconsistent z-values ​​of the cloud coordinates of two adjacent pixels. Therefore, in the adjacent regions of the edge of the largest connected domain, the average value between the coordinates of adjacent point clouds is calculated, and the average value is used to replace the original z value to achieve point cloud smoothing; therefore, in the adjacent regions, the z values ​​of two adjacent point cloud coordinates are selected in a loop, z1 (i.e., the first depth value corresponding to the first interpolation coordinate) and z2 (i.e., the second depth value corresponding to the second interpolation coordinate) are calculated for deviation; a deviation threshold is set, and in this embodiment, the deviation threshold is set to 2mm; the absolute value of the difference between z1 and z2 (i.e., the absolute value of the deviation) is calculated, i.e., abs(z1-z2); when abs(z1-z2) is less than 2mm, z1 and z2 are substituted into formula (7) respectively, i.e. Then, the disparities d1 (i.e., the first smoothed disparity value) and d2 (i.e., the second smoothed disparity value) corresponding to z1 and z2 are calculated respectively; the average value of d1 and d2 is calculated, i.e., d1 + d2 / 2 = d3 (i.e., the average disparity value); correspondingly, d3 is used to replace the original z1 and z2 respectively, thereby achieving point cloud smoothing; the fusion template diagram is shown below. Figure 7 As shown, Figure 7 The color difference section is a blending template.

[0124] Example 2

[0125] This embodiment is based on the same inventive concept as the single / binocular point cloud interpolation method described in Embodiment 1, and provides a single / binocular point cloud interpolation system, such as... Figure 8 As shown, it includes: a single-target positioning module, a binocular point cloud computing module, and a point cloud interpolation module;

[0126] In the single- and binocular point cloud interpolation system, the single-target calibration module is used to configure the camera system and the first reference object; the single-target calibration module is also used to set the camera reference plane and the first phase-to-height mapping formula; the single-target calibration module performs a single-target calibration operation based on the camera system, the first reference object, the camera reference plane and the first phase-to-height mapping formula to obtain the phase-to-height mapping formula to be referenced and the rotation and translation matrix;

[0127] Specifically, the camera system includes a monocular system and a binocular system; the binocular system is based on the monocular system; the camera reference plane includes: a first reference plane, a second reference plane, a third reference plane, and a fourth reference plane;

[0128] Specifically, the single-target calibration operation includes: the single-target calibration module sets the first reference plane as the camera reference plane; the single-target calibration module performs a difference calculation step based on the second reference plane, the third reference plane, the fourth reference plane, and the camera reference plane to obtain first difference information, second difference information, and third difference information; the single-target calibration module substitutes the first difference information into the first phase-to-height mapping relationship to obtain a first height relationship; the single-target calibration module substitutes the second difference information into the first phase-to-height mapping relationship to obtain a second height relationship; the single-target calibration module substitutes the third difference information into the first phase-to-height mapping relationship to obtain a third height relationship; the single-target calibration module identifies a first constant in the first phase-to-height mapping relationship and calculates the constant value corresponding to the first constant by combining the first height relationship, the second height relationship, and the third height relationship; single... The target calibration module updates the first constant in the first phase-to-height mapping relationship based on the constant value to obtain the phase-to-height mapping relationship to be referenced; the single-target calibration module sets the first pixel of the monocular system; the single-target calibration module obtains the first reference phase of the first pixel relative to the first reference object based on the monocular system; the single-target calibration module obtains the second reference phase of the first pixel relative to the first reference plane based on the monocular system; the single-target calibration module obtains the third reference phase of the first pixel relative to the second reference plane based on the monocular system; the single-target calibration module performs a reference plane determination step based on the first reference phase, the second reference phase, and the third reference phase to obtain the first reference plane to be calibrated; the single-target calibration module performs single-target calibration on the monocular system using a phase-to-height mapping method based on the phase-to-height mapping relationship to be referenced and the first reference plane to be calibrated to obtain the rotation-translation matrix.

[0129] Specifically, the difference calculation steps include: the single-target positioning module sets the second pixel of the monocular system; the single-target positioning module obtains the first planar phase of the second pixel with respect to the camera reference plane based on the monocular system; the single-target positioning module obtains the second planar phase of the second pixel with respect to the second reference plane based on the monocular system; the single-target positioning module obtains the third planar phase of the second pixel with respect to the third reference plane based on the monocular system; the single-target positioning module obtains the fourth planar phase of the second pixel with respect to the fourth reference plane based on the monocular system; the single-target positioning module calculates the first phase difference between the second planar phase and the first planar phase; the single-target positioning module calculates the second phase difference between the third planar phase and the first planar phase; the single-target positioning module calculates the third phase difference between the fourth planar phase and the first planar phase; the single-target positioning module, based on the monocular system... The system acquires a first height value for the first reference plane; a single-target positioning module acquires a second height value for the second reference plane based on the monocular system; a single-target positioning module acquires a third height value for the third reference plane based on the monocular system; a single-target positioning module acquires a fourth height value for the fourth reference plane based on the monocular system; a single-target positioning module calculates a first height difference between the second height value and the first height value; a single-target positioning module calculates a second height difference between the third height value and the first height value; a single-target positioning module calculates a third height difference between the fourth height value and the first height value; a single-target positioning module integrates the first phase difference value and the first height difference value to obtain first difference information; a single-target positioning module integrates the second phase difference value and the second height difference value to obtain second difference information; a single-target positioning module integrates the third phase difference value and the third height difference value to obtain third difference information.

[0130] Specifically, the reference plane determination step includes: a single-target calibration module calculating a first absolute value of the difference between the first reference phase and the second reference phase; a single-target calibration module calculating a second absolute value of the difference between the first reference phase and the third reference phase; a single-target calibration module comparing the first absolute value and the second absolute value, and determining whether the first absolute value is greater than the second absolute value; if yes, the single-target calibration module selects the first reference plane as the first reference plane to be calibrated; if no, the single-target calibration module selects the second reference plane as the first reference plane to be calibrated.

[0131] In the single / binocular point cloud interpolation system, the binocular point cloud computing module is used to configure the binocular calibration method and set the binocular point cloud formula group; the binocular point cloud computing module performs binocular point cloud computing operations based on the camera system, the binocular calibration method and the binocular point cloud formula group to obtain binocular calibration information, the first binocular point cloud and the phase at the hole;

[0132] Specifically, the binocular calibration method is the Zhang Zhengyou calibration method; the binocular point cloud formula group is configured with: a first x-value calculation formula, a first y-value calculation formula, and a first z-value calculation formula;

[0133] Specifically, the binocular point cloud computing operation includes: the binocular point cloud computing module using the Zhang Zhengyou calibration method to perform binocular calibration on the binocular system, obtaining a first focal length parameter, a first baseline parameter, pixel center information, and pixel size information; the binocular point cloud computing module integrating the first focal length parameter, the first baseline parameter, the pixel center information, and the pixel size information to obtain the binocular calibration information; the binocular point cloud computing module selecting a first camera and a second camera in the binocular system; the binocular point cloud computing module acquiring several first phases to be selected based on the first camera; the binocular point cloud computing module acquiring several second phases to be selected based on the second camera; and the binocular point cloud computing module performing a corresponding point matching step based on the several first phases to be selected and the several second phases to be selected.

[0134] Specifically, the corresponding point matching step includes: the binocular point cloud computing module selecting a first corresponding phase from a plurality of first phases to be screened; the binocular point cloud computing module selecting a second corresponding phase from a plurality of second phases to be screened that matches the phase value of the first corresponding phase; the binocular point cloud computing module identifying the first phase coordinates corresponding to the first corresponding phase and the second phase coordinates corresponding to the second corresponding phase; the binocular point cloud computing module identifying the first row value and the first column value in the first phase coordinates; identifying the second row value in the second phase coordinates; the binocular point cloud computing module calculating the difference between the second row value and the first row value to obtain a first corresponding disparity value; after the binocular point cloud computing module executes the corresponding point matching step, the binocular point cloud computing module will... The first x-value calculation formula is obtained by substituting the row value and the first focal length parameter into the first x-value calculation formula; the second x-value calculation formula is obtained by substituting the first column value and the first focal length parameter into the first y-value calculation formula; the third z-value calculation formula is obtained by substituting the first focal length parameter, the first baseline parameter, and the first corresponding disparity value into the first z-value calculation formula; the fourth z-value calculation formula is obtained by substituting the z-value into the second x-value calculation formula and the second y-value calculation formula respectively; the fifth z-value calculation formula is obtained by substituting the z-value into the second x-value calculation formula and the second y-value calculation formula respectively; the sixth z-value calculation formula is obtained by generating binocular point cloud coordinates based on the x-value, y-value, and z-value; and the seventh z-value calculation formula is obtained by constructing the first binocular point cloud based on the binocular point cloud coordinates.

[0135] Specifically, the binocular point cloud computing operation further includes: when the binocular point cloud computing module performs the same-name point matching step, it determines whether there is a second same-name phase among a plurality of second phases to be screened that matches the phase value of the first same-name phase. If not, the binocular point cloud computing module sets the first same-name phase as the phase at the hole.

[0136] In the single / binocular point cloud interpolation system, the point cloud interpolation module is used to set a set of point cloud interpolation calculation formulas; the point cloud interpolation module performs point cloud interpolation operations based on the camera reference plane, the phase-to-height mapping relationship to be referenced, the rotation and translation matrix, the binocular positioning information, the first binocular point cloud, the phase at the hole, and the set of point cloud interpolation calculation formulas.

[0137] Specifically, the point cloud interpolation calculation formula group includes: monocular point cloud computing formula and point cloud interpolation coordinate calculation formula;

[0138] Specifically, the point cloud interpolation operation includes: the point cloud interpolation module sets the first camera or the second camera corresponding to the phase at the hole as the reference camera; the point cloud interpolation module identifies the first hole pixel corresponding to the phase at the hole in the reference camera; the point cloud interpolation module obtains the first hole judgment phase of the first hole pixel relative to the first reference plane based on the reference camera; the point cloud interpolation module obtains the second hole judgment phase of the first hole pixel relative to the second reference plane based on the reference camera; the point cloud interpolation module performs the reference plane determination step based on the hole phase, the first hole judgment phase, and the second hole judgment phase to obtain the first interpolation calculation reference plane; the point cloud interpolation module sets the first hole judgment phase or the second hole judgment phase corresponding to the first interpolation calculation reference plane as the first phase to be calculated; the point cloud interpolation module calculates the difference between the hole phase and the first phase to be calculated to obtain the hole phase variable; the point cloud interpolation module substitutes the hole phase variable into the mapping relationship between the phase to be referenced and the height to obtain the height difference at the hole; the point cloud interpolation module sets... The height difference at the hole is defined as the z-coordinate value of the monocular point cloud; the point cloud interpolation module sets the image coordinate calculation formula based on the dual-target positioning information; the point cloud interpolation module sets the pixel coordinate system corresponding to the reference camera based on the dual-target positioning information; the point cloud interpolation module obtains the hole pixel coordinates of the pixel at the first hole in the pixel coordinate system; the point cloud interpolation module substitutes the hole pixel coordinates into the image coordinate calculation formula to obtain the image coordinates at the hole; the point cloud interpolation module substitutes the image coordinates at the hole into the monocular point cloud computing formula to obtain the monocular point cloud to be referenced. The point cloud interpolation module substitutes the coordinate relationship of the monocular point cloud to be referenced and the rotation and translation matrix into the point cloud interpolation coordinate calculation formula to obtain the monocular point cloud x-coordinate value and monocular point cloud y-coordinate value; the point cloud interpolation module substitutes the monocular point cloud z-coordinate value, the monocular point cloud x-coordinate value and the monocular point cloud y-coordinate value into the point cloud interpolation coordinate calculation formula to obtain the point cloud interpolation coordinate; the point cloud interpolation module replaces the binocular point cloud coordinates corresponding to the phase at the hole in the first binocular point cloud with the point cloud interpolation coordinate to obtain a complete binocular point cloud.

[0139] Specifically, it also includes: a point cloud interpolation module performing point cloud smoothing processing based on the first z-value calculation formula, the phase at the hole, and the complete binocular point cloud; the point cloud smoothing processing includes: the point cloud interpolation module confirming a connected region in the complete binocular point cloud based on the phase at the hole; configuring a fusion template; the point cloud interpolation module setting an expanded adjacent region corresponding to the connected region based on the fusion template; identifying a first interpolation coordinate and a second interpolation coordinate in the expanded adjacent region; the point cloud interpolation module identifying a first depth value corresponding to the first interpolation coordinate; the point cloud interpolation module identifying a second depth value corresponding to the second interpolation coordinate; the point cloud interpolation module... A deviation threshold is set, and the absolute value of the deviation between the first depth value and the second depth value is calculated. The point cloud interpolation module determines whether the absolute value of the deviation is less than the deviation threshold. If it is less, the point cloud interpolation module substitutes the first depth value and the second depth value into the first z-value calculation formula to obtain a first smoothed disparity value and a second smoothed disparity value. The point cloud interpolation module calculates the average disparity value of the first smoothed disparity value and the second smoothed disparity value. The point cloud interpolation module replaces the first depth value in the first interpolation coordinate with the average disparity value. The point cloud interpolation module replaces the second depth value in the second interpolation coordinate with the average disparity value.

[0140] Example 3

[0141] This embodiment provides a computer-readable storage medium, including:

[0142] The storage medium is used to store computer software instructions used to implement the single- and binocular point cloud interpolation method described in Embodiment 1 above. It includes a program for executing the single- and binocular point cloud interpolation method described above. Specifically, the executable program can be built into the single- and binocular point cloud interpolation system described in Embodiment 2. In this way, the single- and binocular point cloud interpolation system can implement the single- and binocular point cloud interpolation method described in Embodiment 1 by executing the built-in executable program.

[0143] Furthermore, the computer-readable storage medium in this embodiment can be any combination of one or more readable storage media, wherein the readable storage medium includes an electrical, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof.

[0144] Unlike existing technologies, the single- and binocular point cloud interpolation method, system, and medium proposed in this application can achieve high-precision interpolation of missing point clouds in binocular 3D measurement, resulting in more complete point cloud reconstruction. Furthermore, this method employs a single- and binocular point cloud fusion architecture for point cloud interpolation and performs point cloud smoothing to generate high-quality point clouds. This method is applicable to different cameras or optical engines, has low limitations, and strong universality. This system provides effective technical support for this method, ultimately overcoming the shortcomings of existing technologies and possessing extremely high application value and forward-looking potential.

[0145] The embodiment numbers disclosed in the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0146] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing related hardware to implement the program, which can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0147] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structural or procedural transformations made based on the content of the present invention's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for interpolating point clouds using both single and binocular cameras, characterized in that, Includes the following steps: Monocular system calibration steps: Configure the camera system and the first reference object; set the camera reference plane and the first phase-to-height mapping formula; perform a single-target calibration operation based on the camera system, the first reference object, the camera reference plane and the first phase-to-height mapping formula to obtain the phase-to-height mapping formula to be referenced and the rotation and translation matrix; The single-target calibration operation includes: setting a first pixel of the monocular system; obtaining a first reference phase of the first pixel relative to the first reference object based on the monocular system; obtaining a second reference phase of the first pixel relative to a first reference plane based on the monocular system; obtaining a third reference phase of the first pixel relative to a second reference plane based on the monocular system; and performing a reference plane determination step based on the first reference phase, the second reference phase, and the third reference phase to obtain a first reference plane to be calibrated. The reference plane determination step includes: calculating a first absolute value of the difference between the first reference phase and the second reference phase; calculating a second absolute value of the difference between the first reference phase and the third reference phase; comparing the first absolute value and the second absolute value to determine whether the first absolute value is greater than the second absolute value; if so, selecting the first reference plane as the first reference plane to be calibrated; if not, selecting the second reference plane as the first reference plane to be calibrated. Binocular point cloud computing steps: Configure a binocular positioning method; set a binocular point cloud formula group; perform binocular point cloud computing operation based on the camera system, the binocular positioning method and the binocular point cloud formula group to obtain binocular positioning information, a first binocular point cloud and phase at the hole; Binocular point cloud interpolation steps: Set a set of point cloud interpolation calculation formulas; perform point cloud interpolation operations based on the camera reference plane, the phase-to-height mapping relationship to be referenced, the rotation and translation matrix, the binocular positioning information, the first binocular point cloud, the phase at the hole, and the set of point cloud interpolation calculation formulas.

2. The method for single / binocular point cloud interpolation according to claim 1, characterized in that: The camera system includes a monocular system and a binocular system; the binocular system is based on the monocular system; the camera reference plane includes: a first reference plane, a second reference plane, a third reference plane, and a fourth reference plane; The binocular calibration method is the Zhang Zhengyou calibration method; the binocular point cloud formula group is configured with: the first x-value calculation formula, the first y-value calculation formula, and the first z-value calculation formula; The point cloud interpolation calculation formula group includes: monocular point cloud computing formula and point cloud interpolation coordinate calculation formula.

3. The method for single / binocular point cloud interpolation according to claim 2, characterized in that: The single-target calibration operation further includes: The first reference plane is set as the camera reference plane. Based on the second reference plane, the third reference plane, the fourth reference plane and the camera reference plane, a difference calculation step is performed to obtain the first difference information, the second difference information and the third difference information. Substituting the first difference information into the first phase-height mapping formula yields the first height formula; substituting the second difference information into the first phase-height mapping formula yields the second height formula; substituting the third difference information into the first phase-height mapping formula yields the third height formula. Identify the first constant in the first phase-to-height mapping formula, and calculate the constant value corresponding to the first constant by combining the first height formula, the second height formula, and the third height formula; update the first constant in the first phase-to-height mapping formula based on the constant value to obtain the phase-to-height mapping formula to be referenced. Based on the phase-height mapping relationship to be referenced and the first calibration reference surface, the monocular system is calibrated using the phase-height mapping method to obtain the rotation and translation matrix.

4. The method for single / binocular point cloud interpolation according to claim 3, characterized in that: The difference calculation steps include: Define a second pixel in the monocular system; obtain a first planar phase of the second pixel with respect to the camera reference plane based on the monocular system; obtain a second planar phase of the second pixel with respect to the second reference plane based on the monocular system; obtain a third planar phase of the second pixel with respect to the third reference plane based on the monocular system; obtain a fourth planar phase of the second pixel with respect to the fourth reference plane based on the monocular system. Calculate the first phase difference between the second plane phase and the first plane phase; calculate the second phase difference between the third plane phase and the first plane phase; calculate the third phase difference between the fourth plane phase and the first plane phase; The monocular system is used to obtain a first height value of the first reference plane; the monocular system is used to obtain a second height value of the second reference plane; the monocular system is used to obtain a third height value of the third reference plane; and the monocular system is used to obtain a fourth height value of the fourth reference plane. Calculate the first height difference between the second height value and the first height value; calculate the second height difference between the third height value and the first height value; calculate the third height difference between the fourth height value and the first height value; Integrate the first phase difference and the first height difference to obtain the first difference information; integrate the second phase difference and the second height difference to obtain the second difference information; integrate the third phase difference and the third height difference to obtain the third difference information.

5. The method for single / binocular point cloud interpolation according to claim 4, characterized in that: The binocular cloud computing operation includes: The binocular system is calibrated using the Zhang Zhengyou calibration method to obtain a first focal length parameter, a first baseline parameter, pixel center information, and pixel size information; the first focal length parameter, the first baseline parameter, the pixel center information, and the pixel size information are integrated to obtain the binocular calibration information; In the binocular system, a first camera and a second camera are selected; based on the first camera, a plurality of first phases to be screened of the first reference object are acquired; based on the second camera, a plurality of second phases to be screened of the first reference object are acquired; based on the plurality of first phases to be screened and the plurality of second phases to be screened, a corresponding point matching step is performed; The corresponding point matching step includes: selecting a first corresponding phase from a plurality of first phases to be screened; selecting a second corresponding phase from a plurality of second phases to be screened that matches the phase value of the first corresponding phase; identifying the first phase coordinates corresponding to the first corresponding phase and the second phase coordinates corresponding to the second corresponding phase; identifying the first row value and the first column value in the first phase coordinates; identifying the second row value in the second phase coordinates; calculating the difference between the second row value and the first row value to obtain a first corresponding disparity value; After performing the corresponding point matching step, the first row value and the first focal length parameter are substituted into the first x-value calculation formula to obtain the second x-value calculation formula; the first column value and the first focal length parameter are substituted into the first y-value calculation formula to obtain the second y-value calculation formula; the first focal length parameter, the first baseline parameter, and the first corresponding disparity value are substituted into the first z-value calculation formula to obtain the z-coordinate value; the z-coordinate value is then substituted into the second x-value calculation formula and the second y-value calculation formula to obtain the x-coordinate value and the y-coordinate value, respectively. Generate stereo point cloud coordinates based on the x-coordinate value, the y-coordinate value, and the z-coordinate value; construct the first stereo point cloud based on the stereo point cloud coordinates.

6. The method for single / binocular point cloud interpolation according to claim 5, characterized in that: The binocular cloud computing operation also includes: When performing the corresponding point matching step, it is determined whether there is a second corresponding phase among the several second phases to be screened that matches the phase value of the first corresponding phase. If there is no second corresponding phase, the first corresponding phase is set as the phase at the hole.

7. The method for single / binocular point cloud interpolation according to claim 6, characterized in that: The point cloud interpolation operation includes: The first camera or the second camera corresponding to the phase at the hole is set as the reference camera; the first hole pixel corresponding to the phase at the hole in the reference camera is identified; the first hole judgment phase of the first hole pixel relative to the first reference plane is obtained based on the reference camera; the second hole judgment phase of the first hole pixel relative to the second reference plane is obtained based on the reference camera; the reference plane determination step is performed based on the hole phase, the first hole judgment phase, and the second hole judgment phase to obtain the first interpolation calculation reference plane; Set the first hole judgment phase or the second hole judgment phase corresponding to the first interpolation calculation reference plane as the first phase to be calculated; calculate the difference between the phase at the hole and the first phase to be calculated to obtain the phase variable at the hole; substitute the phase variable at the hole into the mapping relationship between the phase to be referenced and the height to obtain the height difference at the hole; set the height difference at the hole as the z-coordinate value of the monocular point cloud; Based on the dual-target positioning information, an image coordinate calculation formula is set; based on the dual-target positioning information, a pixel coordinate system corresponding to the reference camera is set; the hole pixel coordinates of the first hole in the pixel coordinate system are obtained; the hole pixel coordinates are substituted into the image coordinate calculation formula to obtain the hole image coordinates; the hole image coordinates are substituted into the monocular point cloud computing formula to obtain the monocular point cloud coordinate relationship to be referenced. Substituting the monocular point cloud coordinate relationship to be referenced and the rotation and translation matrix into the point cloud interpolation coordinate calculation formula, we obtain the monocular point cloud x-coordinate value and the monocular point cloud y-coordinate value; substituting the monocular point cloud z-coordinate value, the monocular point cloud x-coordinate value and the monocular point cloud y-coordinate value into the point cloud interpolation coordinate calculation formula, we obtain the point cloud interpolation coordinates. In the first binocular point cloud, the binocular point cloud coordinates corresponding to the phase at the hole are replaced with the point cloud interpolation coordinates to obtain a complete binocular point cloud.

8. The method for single / binocular point cloud interpolation according to claim 7, characterized in that: The single / binocular point cloud interpolation method further includes: performing point cloud smoothing processing based on the first z-value calculation formula, the phase at the hole, and the complete binocular point cloud; The point cloud smoothing process includes: Based on the phase at the hole, a connected region is confirmed in the complete binocular point cloud; a fusion template is configured; based on the fusion template, an expanded adjacent region corresponding to the connected region is set; a first interpolation coordinate and a second interpolation coordinate are identified in the expanded adjacent region; a first depth value corresponding to the first interpolation coordinate is identified; and a second depth value corresponding to the second interpolation coordinate is identified. Set a deviation threshold, calculate the absolute value of the deviation between the first depth value and the second depth value; determine whether the absolute value of the deviation is less than the deviation threshold; if it is less, substitute the first depth value and the second depth value into the first z-value calculation formula respectively to obtain the first smoothed disparity value and the second smoothed disparity value. Calculate the average disparity value of the first smoothed disparity value and the second smoothed disparity value; replace the first depth value in the first interpolation coordinates with the average disparity value; replace the second depth value in the second interpolation coordinates with the average disparity value.

9. A single / binocular point cloud interpolation system based on the single / binocular point cloud interpolation method according to any one of claims 1 to 8, characterized in that, include: Single-target positioning module, binocular point cloud computing module, and point cloud interpolation module; The single-target calibration module is used to configure the camera system and the first reference object; the single-target calibration module is also used to set the camera reference plane and the first phase-to-height mapping formula; the single-target calibration module performs a single-target calibration operation based on the camera system, the first reference object, the camera reference plane and the first phase-to-height mapping formula to obtain the phase-to-height mapping formula to be referenced and the rotation and translation matrix; The binocular point cloud computing module is used to configure the binocular positioning method and set the binocular point cloud formula group; the binocular point cloud computing module performs binocular point cloud computing operation based on the camera system, the binocular positioning method and the binocular point cloud formula group to obtain binocular positioning information, the first binocular point cloud and the phase at the hole; The point cloud interpolation module is used to set a set of point cloud interpolation calculation formulas; the point cloud interpolation module performs point cloud interpolation operations based on the camera reference plane, the phase-to-height mapping relationship to be referenced, the rotation and translation matrix, the binocular positioning information, the first binocular point cloud, the phase at the hole, and the set of point cloud interpolation calculation formulas.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the single- and binocular point cloud interpolation method according to any one of claims 1 to 8.

Images