A data fusion method based on structured light

Abstract

The present invention discloses a data fusion method based on structured light, comprising: S1: configuring a structured light vision system including two monocular systems and one binocular system, and calibrating the internal and external parameters of the monocular system and the binocular system; S2: calibrating using a calibration plane to obtain a pixel correction table to be matched on a projector plane, and moving the calibration plane to different positions to obtain n pixel correction tables, where n is 8 to 15; S3: performing binocular reconstruction, and in the reconstruction process, identifying the points of visual field occlusion and matching failure in the binocular system; S4: for the points of visual field occlusion and matching failure in the binocular system in step S3, using the first monocular system and / or the second monocular system, combined with the n pixel correction tables obtained in step S2, to perform monocular reconstruction. The present invention can meet the general accuracy requirements of monocular and binocular data fusion based on structured light.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a data fusion method based on structured light. Background Art

[0002] Three-dimensional measurement based on structured light has the advantages of non-contact, high precision and high efficiency. Its measurement solutions are mainly monocular reconstruction of projector + single camera and binocular reconstruction of projector + dual cameras. Binocular reconstruction based on structured light has high precision and good effect, but it will be affected by binocular field of view occlusion, camera performance differences and light source direction, resulting in missing reconstruction points. Monocular reconstruction based on structured light is fast, but the reconstruction accuracy is poor.

[0003] Specifically, during reconstruction, binocular reconstruction based on structured light will result in missing features of the projected pattern and occlusion of the monocular field of view due to the relatively smooth surface or large undulations of the object to be measured, which will lead to matching failure and obvious missing reconstruction points; monocular reconstruction based on structured light may not be affected by the direction of the light source and the field of view to a certain extent during reconstruction, but since the calibration of the projector needs to be completed by the camera, and the camera calibration itself has certain errors, the errors are accumulated in the calibration results of the projector, making the reconstruction results of this scheme worse. In the monocular and binocular combination scheme based on structured light, if the monocular and binocular reconstruction data are directly fused, there will be a certain deviation between the binocular reconstruction results and the monocular reconstruction results, which will greatly affect the reconstruction quality.

[0004] The disclosure of the above background technology content is only used to assist in understanding the concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0005] In view of the above technical problems, the present invention proposes a data fusion method based on structured light, which can meet the general accuracy requirements of monocular and binocular data fusion based on structured light.

[0006] In order to achieve the above object, the present invention adopts the following technical solutions:

[0007] An embodiment of the present invention discloses a data fusion method based on structured light, comprising the following steps:

[0008] S1: configuring a structured light vision system including two monocular systems and one binocular system, and calibrating internal and external parameters of the monocular system and the binocular system, wherein the structured light vision system includes a first camera, a projector, and a second camera, the first camera and the projector constitute a first monocular system, the second camera and the projector constitute a second monocular system, and the first camera, the projector, and the second camera together constitute a binocular system;

[0009] S2: calibrating with the calibration plane to obtain a pixel correction table to be matched on the projector plane, and moving the calibration plane to different positions to obtain n pixel correction tables, where n is 8 to 15;

[0010] S3: Perform binocular reconstruction, and during the reconstruction process, identify points of visual field obstruction and matching failure in the binocular system;

[0011] S4: For the points with blocked field of view and failed matching in the binocular system in step S3, monocular reconstruction is performed using the first monocular system and / or the second monocular system in combination with the n pixel correction tables obtained in step S2, and the result of the monocular reconstruction is fused with the result of the binocular reconstruction.

[0012] Preferably, step S2 specifically includes:

[0013] S21: placing a white calibration plane at a position within the measurement depth range, projecting a grating with the projector, and performing binocular reconstruction of the first camera and the second camera according to the grating phase;

[0014] S22: According to the internal and external parameters of the projector calibrated in advance in step S1, the points reconstructed by binocular in step S21 are mapped into the plane of the projector to obtain the actual value of the projector pixel coordinate;

[0015] S23: Calculate the deviation between the theoretical value of the projector pixel coordinate based on phase matching and the actual value of the projector pixel coordinate obtained in step S22 to obtain the pixel correction value of the point;

[0016] S24: traverse all points of binocular reconstruction, each point corresponds to a pixel correction amount, and store the value of the pixel correction amount corresponding to each point in a pixel correction table;

[0017] S25: Move the white calibration plane within the measurement range according to a preset interval, and repeat steps S21 to S24 to obtain n pixel correction tables of different depths.

[0018] Preferably, step S22 specifically includes:

[0019] According to the calibration parameters obtained in step S1 and the binocular reconstruction result obtained in step S21, the image pixel coordinates Pl (u l ,v l )、Pr(u r ,v r ) and the actual value of the projector pixel coordinates:

[0020]

[0021] Where: Zc is the focal length of the device corresponding to the pixel coordinate (u, v), the device is the first camera, the projector or the second camera, [m ij ] 3×4 is the internal and external parameter matrix of the monocular system and the binocular system, (X W ,Y W ,Z W ) is the three-dimensional coordinate in the world coordinate system obtained by binocular reconstruction.

[0022] Preferably, step S23 specifically includes:

[0023] Based on the phase solution results of the first camera and the second camera, the theoretical values ​​of the projector pixel coordinates of the monocular system corresponding to the first camera and the second camera are calculated by the following formula, and then the deviation between the theoretical value and the actual value of the pixel coordinates is calculated to obtain the corresponding pixel correction amount:

[0024]

[0025] Where: T represents the number of pixels occupied by one cycle when the phase is not unwrapped, φ V Indicates the absolute phase of the vertical stripes corresponding to the current camera pixel coordinates, φ H Indicates the absolute phase of the horizontal stripe corresponding to the current camera pixel coordinates; (u, v) is the theoretical value of the projector pixel coordinates of the monocular system.

[0026] Preferably, step S3 specifically includes:

[0027] S31: Under actual scene scanning conditions, performing phase decoding on the grating images captured by the first camera and the second camera, and performing phase homonymous point matching;

[0028] S32: Find points with valid phases and the same phase name and points where matching fails in the first view and the second view, and calculate theoretical values ​​of projector pixel coordinates corresponding to these points according to the phases, wherein the first view is a grating image captured by the first camera, and the second view is a grating image captured by the second camera, wherein the valid phase refers to the phase calculated from pixel points that meet preset contrast requirements.

[0029] Preferably, calculating the corresponding projector pixel coordinates according to the phase in step S32 specifically includes: calculating the corresponding projector pixel coordinate theoretical value using the following formula:

[0030]

[0031] Where: T represents the number of pixels occupied by one cycle when the phase is not unwrapped, φ V Indicates the absolute phase of the vertical stripes corresponding to the current camera pixel coordinates, φ H Indicates the absolute phase of the horizontal stripe corresponding to the current camera pixel coordinates; (u, v) is the theoretical value of the projector pixel coordinates of the monocular system.

[0032] Preferably, step S4 specifically includes:

[0033] S41: performing monocular reconstruction according to the projector pixel coordinates and the single camera pixel coordinates;

[0034] S42: performing interpolation compensation of pixel coordinates in a pixel correction table on the monocular reconstruction result to correct the projector pixel coordinates;

[0035] S43: performing monocular reconstruction of spatial points again on the corrected projector pixel coordinates, and fusing the reconstruction result with the binocular reconstruction result based on structured light.

[0036] Preferably, step S42 specifically includes: first, finding the i-th and i+1-th pixel correction tables and corresponding pixel compensation amounts corresponding to the depth range according to the result of monocular reconstruction, and then performing linear interpolation calculation according to the following formula to finally obtain the correction compensation amount (Δu, Δv) of the projector pixel coordinates:

[0037]

[0038]

[0039] Where, d i d i+1 is the depth range threshold of the i-th and i+1-th pixel correction tables, (Δu i ,Δv i ) is the pixel compensation amount of the i-th pixel correction table, (Δu i+1 ,Δv i+1 ) is the pixel compensation amount of the i+1th pixel correction table, and z is the current depth value.

[0040] Preferably, step S43 specifically includes: obtaining the error compensation amount (Δu, Δv) of the projector pixel coordinates at the current depth, and then solving the following equation again by (u', v') and the camera coordinates under the corresponding monocular system to perform monocular reconstruction, and obtaining the corrected monocular reconstructed three-dimensional coordinates:

[0041]

[0042] Where (X, Y, Z) is the 3D coordinate of the monocular reconstruction to be solved, (u1, v1) and (u2, v2) are the camera pixel coordinates and projector pixel coordinates Pp respectively, [m ij 1 ] 3×4 、[m ij 2 ] 3×4 are the intrinsic and extrinsic parameter matrices of the camera and the projector respectively; wherein the camera in the monocular system is the first camera or the second camera.

[0043] Another embodiment of the present invention discloses a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the steps of the above-mentioned data fusion method.

[0044] Compared with the prior art, the beneficial effect of the present invention lies in: the monocular and binocular data fusion method based on structured light provided by the present invention comprehensively utilizes the respective advantages of the monocular reconstruction based on structured light and the binocular reconstruction system based on structured light, reconstructs the main part by calibrating the projector pixel coordinate pixel correction table in advance, and compensates for the monocular reconstruction result after error correction for the missing part of the reconstruction, thereby ensuring the accuracy of the binocular reconstruction to a great extent, and making full use of the advantages of fast monocular reconstruction speed and good scanning integrity, so that the three-dimensional reconstruction based on structured light can not only ensure the scanning accuracy but also improve the integrity of the three-dimensional reconstruction, and does not significantly increase the time consumption, and can meet the current industrial scanning needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] Figure 1 It is a flow chart of a monocular and binocular data fusion method based on structured light disclosed in an embodiment of the present invention.

[0046] Figure 2 It is a schematic diagram of the structural configuration of the monocular system and the binocular system of the present invention.

[0047] Figure 3 It is a schematic diagram of the configuration of the calibration plane of the present invention at different distances.

[0048] Figure 4 It is an overall algorithm flow chart of the monocular and binocular data fusion method based on structured light in a preferred embodiment of the present invention.

[0049] Figure 5 It is a schematic diagram of the pixel correction principle of the present invention.

[0050] Explanation of the marks in the accompanying drawings: 1, left camera; 2, projector; 3, right camera; 4, calibration plane; 12, left monocular system; 23, right monocular system; 123, binocular system; L, left camera view; P, projector view; R, right camera view; Pl, left view pixel coordinates; Pp, projector pixel coordinates; Pp', corrected projector pixel coordinates; Pr, right view pixel coordinates; a1, left camera field of view angle; a2, projector field of view angle; a3, right camera field of view angle. DETAILED DESCRIPTION

[0051] The following is a detailed description of the embodiments of the present invention. It should be emphasized that the following description is only exemplary and is not intended to limit the scope and application of the present invention.

[0052] It should be noted that when a component is referred to as being "fixed to" or "disposed on" another component, it can be directly on the other component or indirectly on the other component. When an element is referred to as being "connected to" another element, it can be directly connected to the other element or indirectly connected to the other element. In addition, connection can be used for both fixing and circuit / signal communication.

[0053] It should be understood that the terms "length", "width", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", etc., indicating the orientation or position relationship, are based on the orientation or position relationship shown in the accompanying drawings, and are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation, and therefore cannot be understood as a limitation on the present invention.

[0054] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of the features. In the description of the embodiments of the present invention, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0055] Through research, the inventors found that the existing reconstruction scheme based on structured light cannot take into account the reconstruction speed, reconstruction accuracy and completeness. Therefore, it is necessary to study a new data fusion method based on structured light binocular reconstruction and monocular reconstruction compensation.

[0056] like Figure 1 As shown, an embodiment of the present invention discloses a monocular and binocular data fusion method based on structured light, including four major steps: S1: configuring a structured light vision system including two monocular systems and one binocular system, and calibrating the internal and external parameters of the monocular system and the binocular system, wherein the structured light vision system includes a first camera, a projector, and a second camera, the first camera and the projector constitute a first monocular system, the second camera and the projector constitute a second monocular system, the first camera, the projector, and the second camera together constitute a binocular system, and a specific embodiment can obtain the internal and external parameters of the first camera, the second camera, and the projector through a ten-parameter distortion model calibration; S2: using a calibration plane to calibrate, obtain a pixel correction table to be matched on the projector plane, and move the calibration plane to different positions to obtain n pixel correction tables, and n is 8 to 15; a specific embodiment can use a white calibration plane for calibration Determine, obtain the pixel correction table of the point to be matched on the projector plane, move different positions, and obtain the n*pixel correction table of the projector pixel coordinate Pp; S3: perform binocular reconstruction, and in the reconstruction process, identify the points with blocked field of view and failed matching in the binocular system; the specific embodiment can perform normal binocular reconstruction, and in the reconstruction process, identify the points with blocked field of view and failed matching in the binocular system according to the valid and failed matching pixels in any of the left and right phase images; S4: for the points with blocked field of view and failed matching in the binocular system in step S3, use the first monocular system and / or the second monocular system, combined with the n pixel correction tables obtained in step S2, to perform monocular reconstruction, and fuse the results of monocular reconstruction with the results of binocular reconstruction. The specific embodiment can use the left monocular system or the right monocular system, combined with the pixel correction table, to perform monocular reconstruction for the above-mentioned missing points.

[0057] In a further embodiment, a method is proposed to effectively identify binocular occluded points and unmatched points; by finding pixels containing valid phases and phase homonymous points that fail to match in the phase map captured by the first camera and the second camera, the corresponding projector pixel coordinates Pp are calculated according to the relationship between the phase and the projection pixel coordinates, and three-dimensional reconstruction is performed based on the projector pixel coordinates Pp and the image coordinates containing the valid phase. The reconstruction results are the points with blocked field of view and failed matching in the binocular system.

[0058] In a further embodiment, a correction method is provided, which can reduce the deviation between monocular reconstruction data and binocular reconstruction data to within 0.02mm; first, binocular and monocular reconstruction based on structured light are performed on the white calibration plane at different distances within the measurement depth range; then, at different depths, the projector pixel coordinates Pp are reversely solved for the first camera and projector, and the second camera and projector one by one based on the binocular reconstruction three-dimensional coordinate results; finally, the deviation between the actual value and the theoretical value of the pixel coordinates corresponding to each phase at different depths is calculated, and a pixel correction table is created pixel by pixel.

[0059] In a further embodiment, the created pixel correction table is used for subsequent normal monocular reconstruction compensation. Specifically, the depth range of the three-dimensional coordinates is obtained by the first monocular reconstruction in the real scene, and two pairs of depth extreme values ​​and compensation extreme values ​​of the distance range in the pixel correction table are searched according to the depth range of the monocular reconstruction. Then, the compensation value of the depth range is linearly interpolated based on the result of the monocular reconstruction. The interpolation result is used to correct the projector pixel coordinates Pp of the monocular reconstruction, and then the three-dimensional reconstruction is performed again using the corrected projector pixel coordinates Pp' and the corresponding camera view.

[0060] like Figures 2 to 5 As shown, a preferred embodiment of the present invention provides a monocular and binocular data fusion method based on structured light, comprising the following steps:

[0061] S1: The system obtains the intrinsic and extrinsic parameters of the monocular system and the binocular system 123 through the ten-parameter distortion model calibration;

[0062] like Figure 2 As shown, the monocular system refers to the left monocular system 12 and the right monocular system 23; the left monocular system 12 is composed of a left camera 1 and a projector 2; the right monocular system 23 is composed of a right camera 3 and a projector 2; the capture range of the left camera 1 refers to the left camera field of view angle a1; the capture range of the right camera 3 refers to the right camera field of view angle a3; the capture range of the projector 2 refers to the projector field of view angle a2;.

[0063] Among them, through the ten-parameter distortion model calibration system, when calibrating the system, it is necessary to calibrate the binocular system 123 and the two monocular systems based on structured light respectively, and finally obtain the internal and external parameters of the binocular system 123, the internal and external parameters of the left camera 1 and the projector 2, and the internal and external parameters of the right camera 3 and the projector 2; the two monocular systems based on structured light refer to the left monocular system 12 composed of the left camera 1 and the projector 2, and the right monocular system 23 composed of the right camera 3 and the projector 2.

[0064] S2: Calibrate using the white calibration plane 4 to obtain a pixel correction table of the point to be matched on the projector plane. Move the white calibration plane 4 to different positions to obtain n pixel correction tables (each calibration can obtain a correction table at a corresponding distance, and multiple correction tables are obtained for calibration at multiple distances), where n is 8 to 15, for example, n is 8, 9, 10, 11, 12, 13, 14 or 15; the specific steps are:

[0065] S21: placing a white calibration plane 4 at a position within the measurement depth range, projecting a grating with a projector 2, and performing binocular reconstruction of the left camera 1 and the right camera 3 according to the grating phase;

[0066] like Figure 3 As shown, three-dimensional reconstruction is performed under the binocular system within the measurement depth range; first, the grating stripes are projected by the projector 2, and then the left view L and the right view R of the grating image are captured by the left camera 1 and the right camera 3 respectively, and then phase resolution and phase matching are performed, and finally the three-dimensional coordinates are reconstructed by the triangulation reconstruction principle. Among them, Figure 3 Where d1-d5 represents the configuration distance interval of each calibration plane during calibration, and d1-d5 may be equal or unequal, and is not limited to 5 values.

[0067] S22: According to the internal and external parameters of the projector 2 calibrated in advance in step S1, the points reconstructed by binocular in step S21 are mapped into the plane of the projector 2 to obtain the reverse solution value of the projector pixel coordinate Pp;

[0068] According to the calibration parameters obtained in step S1 and the binocular reconstruction result obtained in step S21, the image pixel coordinates Pl (u l ,v l )、Pr(u r ,v r ) and the actual value of the projector pixel coordinates.

[0069]

[0070] In formula (1), Zc is the focal length of the device corresponding to the pixel coordinate (u, v) (left camera 1, projector 2 or right camera 3), [m ij ] 3×4 is the intrinsic and extrinsic parameter matrix of the corresponding device (including left camera 1, projector 2 and right camera 3), (X W ,Y W ,Z W ) is the three-dimensional coordinates of binocular reconstruction.

[0071] S23: performing deviation calculation on the theoretical value of the projector pixel coordinate based on phase matching and the reverse solution value of the projector pixel coordinate Pp obtained in step S22, thereby obtaining the pixel correction amount of the point;

[0072] Among them, based on the phase solution results of the left camera 1 and the right camera 3, the theoretical values ​​of the projector pixel coordinates of the monocular system corresponding to the left camera 1 and the right camera 3 are calculated by formula (2), and then the deviation between the theoretical value and the actual value of the pixel coordinate is calculated to obtain the corresponding pixel correction amount.

[0073]

[0074] In formula (2), T represents the number of pixels occupied by one cycle when the phase is not unwrapped, φ V Indicates the absolute phase of the vertical stripes corresponding to the current camera pixel coordinates, φ H Indicates the absolute phase of the horizontal stripe corresponding to the current camera pixel coordinates; (u, v) is the theoretical projector pixel coordinates of the monocular system.

[0075] S24: traverse all points of binocular reconstruction, each point corresponds to a pixel correction amount, and store the value of the pixel correction amount corresponding to each point in a pixel correction table;

[0076] Among them, all points of binocular reconstruction are traversed so that each point obtains a pixel correction amount, and these correction amounts are stored as a pixel correction table according to the pixel coordinate relationship of the left camera 1 or the right camera 3.

[0077] S25: Figure 3 As shown, the white calibration plane 4 is moved within the measurement range at a certain interval, thereby obtaining n pixel correction tables of different depths.

[0078] Wherein, within the measuring range, by moving the white calibration plane 4 at a certain interval, steps S21, S22, S23 and S24 are repeated; different distance sizes and placement quantities of the calibration plane 4 are limited according to the needs of actual applications.

[0079] S3: Perform normal binocular reconstruction, and during the reconstruction process, identify points in the binocular system 123 where the visual field is blocked and the matching fails. The specific steps are:

[0080] S31: Under actual scene scanning conditions, the grating images captured by the left camera 1 and the right camera 3 are phase resolved, and phase homonymous point matching is performed;

[0081] Among them, by de-phasing the images captured by the left camera 1 and the right camera 3 under actual scene scanning conditions and performing phase homonymous point matching, the pixel coordinates and their phase values ​​with valid phase values ​​and failed matching in the phase images of the left camera 1 and the right camera 3 are marked.

[0082] S32: Find points with valid phases and the same phase name and pixels with failed matching in the left view L and the right view R, and calculate the corresponding projector pixel coordinates Pp according to the phases, where the valid phase refers to the phase obtained by solving the pixel points that meet the preset contrast requirement;

[0083] According to the image coordinates and phase values ​​of the marker, the projector pixel coordinates Pp in the corresponding monocular system are calculated by formula (2).

[0084] S4: For the missing points in step S3 (including points with visual field obstruction and points with matching failure), the left monocular system 12 or the right monocular system 23 is used in combination with the pixel correction table to perform monocular reconstruction, such as Figure 5 As shown, P0 represents the 3D point reconstructed by binocular with phase matching, and P1 represents the 3D point reconstructed by monocular with phase matching; the specific steps are:

[0085] S41: Perform three-dimensional reconstruction according to the projector pixel coordinates Pp and the single camera pixel coordinates;

[0086] Among them, the three-dimensional coordinates under monocular are obtained by monocularly reconstructing the coordinates with valid phase and missing matching of homonymous points and their corresponding projector pixel coordinates Pp in the monocular system.

[0087] S42: interpolating and compensating the pixel coordinates in the pixel correction table for the monocular reconstruction result. It can be seen from step S25 that the point of the depth value needs to be corrected using the i-th and i+1-th pixel correction tables;

[0088] Among them, in the actual scene, the monocular reconstruction result due to the missing binocular system 123 matching is corrected for the projector pixel Pp; first, according to the result of the monocular reconstruction, the i-th and i+1-th pixel correction tables of the corresponding depth range and their corresponding pixel compensation amounts are found, and then linear interpolation calculation is performed according to formula (3), and finally the correction compensation amount (Δu, Δv) of the projector pixel coordinate Pp is obtained.

[0089]

[0090] In formula (3), d i d i+1 is the depth range threshold of the i-th and i+1-th pixel correction tables, (Δu i ,Δv i ) is the pixel compensation amount of the i-th pixel correction table, (Δui+1 ,Δv i+1 ) is the pixel compensation amount of the i+1th pixel correction table, and z is the current depth value.

[0091] S43: reconstructing the three-dimensional space point of the corrected projector pixel coordinates Pp' again, and fusing the reconstruction result with the binocular reconstruction result based on structured light.

[0092] Among them, the corrected projector pixel coordinates Pp'(u',v') can be obtained by the error compensation amount (Δu, Δv) of the projector pixel coordinates at the current depth. Then, (u',v') and the camera coordinates under the corresponding monocular system are used to solve equation (4) again for monocular reconstruction, and the corrected monocular reconstructed 3D coordinates can be obtained.

[0093]

[0094] In formula (4), (X, Y, Z) is the 3D coordinate of the monocular reconstruction to be solved, (u1, v1) and (u2, v2) are the camera pixel coordinates and projector pixel coordinates Pp respectively, [m ij 1 ] 3×4 、[m ij 2 ] 3×4 are the intrinsic and extrinsic parameter matrices of the camera and projector 2 respectively; at this time, the camera in the monocular system may be either the left camera 1 or the right camera 3.

[0095] Another embodiment of the present invention further provides a computer-readable storage medium, which stores computer-executable instructions. When the computer-executable instructions are called and executed by the processor, the computer-executable instructions prompt the processor to implement the above-mentioned monocular and binocular data fusion method. The specific implementation can be found in the method embodiment, which will not be repeated here.

[0096] The present invention discloses a method for fusion of monocular and binocular data based on structured light. The method is divided into four major steps: first, the internal and external parameters of the left and right cameras and the projector are obtained through calibration of a ten-parameter distortion model; second, a pixel correction table of the points to be matched on the projector plane is obtained by calibrating a white calibration plane, and the calibration plane is moved to different positions to obtain an n* pixel correction table; then, normal binocular reconstruction is performed, and during the reconstruction process, points with blocked field of view and points with failed matching in the binocular system are identified; finally, for the missing points, a left monocular or right monocular system is used in combination with the pixel correction table for monocular reconstruction. The present invention realizes a relatively accurate fusion of monocular and binocular data based on structured light, and meets the general accuracy requirements of monocular and binocular reconstruction.

[0097] Furthermore, the present invention effectively solves the problem of missing binocular reconstruction data. The solution is optimized only from the algorithm level, and no additional equipment is added to achieve the missing points, which effectively controls the cost and system complexity.

[0098] Furthermore, the present invention utilizes observation results from three different angles for reconstruction, which has obvious advantages in the integrity of a single reconstruction. For example, the present system can obtain grooves and deep holes that cannot be scanned by other binocular systems. In addition, the present invention also aims to solve the problem of large errors between monocular reconstruction and binocular reconstruction data, and through pixel correction, the error of monocular and binocular data fusion is controlled within 0.02mm.

[0099] The background section of the present invention may contain background information about the problem or environment of the present invention, rather than describing the prior art by others. Therefore, the content contained in the background section is not an admission by the applicant that the prior art is available.

[0100] The above content is a further detailed description of the present invention in combination with specific / preferred embodiments, and it cannot be determined that the specific implementation of the present invention is limited to these descriptions. For ordinary technicians in the technical field to which the present invention belongs, without departing from the concept of the present invention, it can also make several substitutions or modifications to these described embodiments, and these substitutions or modifications should be regarded as belonging to the protection scope of the present invention. In the description of this specification, the description of reference terms "an embodiment", "some embodiments", "preferred embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in combination with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily target the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, in the absence of mutual contradiction, the technical personnel in this field can combine and combine the different embodiments or examples described in this specification and the features of different embodiments or examples. Although the embodiments of the present invention and their advantages have been described in detail, it should be understood that various changes, substitutions and modifications can be made herein without departing from the scope defined by the attached claims.

Claims

1. A data fusion method based on structured light, characterized in that: The following steps are involved: S1: configuring a structured light vision system including two monocular systems and one binocular system, and calibrating internal and external parameters of the monocular systems and the binocular systems, wherein the structured light vision system includes a first camera, a projector, and a second camera, the first camera and the projector constitute a first monocular system, the second camera and the projector constitute a second monocular system, and the first camera, the projector, and the second camera together constitute a binocular system; S2: calibrating with the calibration plane to obtain a pixel correction table to be matched on the projector plane, and moving the calibration plane to different positions to obtain n pixel correction tables, where n is 8 to 15; S3: Perform binocular reconstruction, and during the reconstruction process, identify points of visual field obstruction and matching failure in the binocular system; S4: for the points with blocked vision and failed matching in the binocular system in step S3, use the first monocular system and / or the second monocular system in combination with the n pixel correction tables obtained in step S2 to perform monocular reconstruction, and fuse the results of the monocular reconstruction with those of the binocular reconstruction; Wherein, step S4 specifically includes: S41: performing monocular reconstruction according to the projector pixel coordinates and the single camera pixel coordinates; S42: performing interpolation compensation of pixel coordinates in a pixel correction table on the monocular reconstruction result to correct the projector pixel coordinates; S43: performing monocular reconstruction of spatial points again on the corrected projector pixel coordinates, and fusing the reconstruction result with the binocular reconstruction result based on structured light; Step S42 specifically includes: first, finding the i-th and i+1-th pixel correction tables and corresponding pixel compensation amounts corresponding to the depth range according to the result of monocular reconstruction, and then performing linear interpolation calculation according to the following formula to obtain the correction compensation amount (Δu, Δv) of the projector pixel coordinates: Where, d i ,d i+1 is the depth range threshold of the i-th and i+1-th pixel correction tables, (Δu i ,Δv i ) is the pixel compensation amount of the i-th pixel correction table, (Δu i+1 ,Δv i+1 ) is the pixel compensation amount of the i+1th pixel correction table, and z is the current depth value.

2. The data fusion method according to claim 1, characterized in that: Step S2 specifically includes: S21: placing a white calibration plane at a position within the measurement depth range, projecting a grating with the projector, and performing binocular reconstruction of the first camera and the second camera according to the grating phase; S22: According to the internal and external parameters of the projector calibrated in advance in step S1, the points reconstructed by binocular in step S21 are mapped into the plane of the projector to obtain the actual value of the projector pixel coordinate; S23: Calculate the deviation between the theoretical value of the projector pixel coordinate based on phase matching and the actual value of the projector pixel coordinate obtained in step S22 to obtain the pixel correction value of the point; S24: traverse all points of binocular reconstruction, each point corresponds to a pixel correction amount, and store the value of the pixel correction amount corresponding to each point in a pixel correction table; S25: Move the white calibration plane within the measurement range according to a preset interval, and repeat steps S21 to S24 to obtain n pixel correction tables of different depths.

3. The data fusion method according to claim 2, characterized in that: Step S22 specifically includes: According to the calibration parameters obtained in step S1 and the binocular reconstruction result obtained in step S21, the image pixel coordinates Pl (u) of the first monocular system and the second monocular system are respectively obtained by the following formula: l ,v l )、Pr(u r ,v r ) and the actual value of the projector pixel coordinates: Where: Zc is the focal length of the device corresponding to the pixel coordinate (u, v), the device is the first camera, the projector or the second camera, [m ij ] 3×4 is the internal and external parameter matrix of the monocular system and the binocular system, (X W ,Y W ,Z W ) is the three-dimensional coordinate in the world coordinate system obtained by binocular reconstruction.

4. The data fusion method according to claim 2, characterized in that: Step S23 specifically includes: Based on the phase solution results of the first camera and the second camera, the theoretical values ​​of the projector pixel coordinates of the monocular system corresponding to the first camera and the second camera are calculated by the following formula, and then the deviation between the theoretical value and the actual value of the pixel coordinates is calculated to obtain the corresponding pixel correction amount: Where: T represents the number of pixels occupied by one cycle when the phase is not unwrapped, φ V Indicates the absolute phase of the vertical stripes corresponding to the current camera pixel coordinates, φ H Indicates the absolute phase of the horizontal stripe corresponding to the current camera pixel coordinates; (u, v) is the theoretical value of the projector pixel coordinates of the monocular system.

5. The data fusion method according to claim 1, characterized in that: Step S3 specifically includes: S31: Under actual scene scanning conditions, performing phase decoding on the grating images captured by the first camera and the second camera, and performing phase homonymous point matching; S32: Find points with valid phases and the same phase names and points with failed matching in the first view and the second view, and calculate theoretical values ​​of projector pixel coordinates corresponding to these points according to the phases, wherein the first view is a grating image captured by the first camera, and the second view is a grating image captured by the second camera, wherein the valid phase refers to the phase calculated from pixel points that meet preset contrast requirements.

6. The data fusion method according to claim 5, characterized in that: Calculating the corresponding projector pixel coordinates according to the phase in step S32 specifically includes: calculating the corresponding projector pixel coordinate theoretical value using the following formula: Where: T represents the number of pixels occupied by one cycle when the phase is not unwrapped, φ V Indicates the absolute phase of the vertical stripes corresponding to the current camera pixel coordinates, φ H Indicates the absolute phase of the horizontal stripe corresponding to the current camera pixel coordinates; (u, v) is the theoretical value of the projector pixel coordinates of the monocular system.

7. The data fusion method according to claim 1, characterized in that: In step S43, the monocular reconstruction of the spatial point of the corrected projector pixel coordinates again includes: the corrected projector pixel coordinates (u', v') can be obtained by obtaining the error compensation amount (Δu, Δv) of the projector pixel coordinates at the current depth, and then the monocular reconstruction is performed by solving the following equation again through (u', v') and the camera coordinates under the corresponding monocular system to obtain the corrected monocular reconstructed three-dimensional coordinates: Where (X, Y, Z) is the 3D coordinate of the monocular reconstruction to be solved, (u1, v1) and (u2, v2) are the camera pixel coordinates and projector pixel coordinates Pp respectively, [m ij 1 ] 3×4 、[m ij 2 ] 3×4 are the intrinsic and extrinsic parameter matrices of the camera and the projector respectively; wherein the camera in the monocular system is the first camera or the second camera.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions. When the computer-executable instructions are called and executed by a processor, the computer-executable instructions prompt the processor to implement the steps of the data fusion method according to any one of claims 1 to 7.

Images