Projector-assisted binocular vision system calibration method and related device

Abstract

The invention discloses a binocular vision system calibration method assisted by a projector, a binocular vision system calibration device, binocular vision system calibration equipment and a computer readable storage medium, and the method comprises the steps: carrying out calibration space calculation based on internal and external parameters calculated by a Zhang's calibration method, and obtaining a calibration space; calculating a target frame for guiding the placement of the calibration plate based on the size of the calibration plate and each calibration space; after the calibration plate is placed, controlling a projector to project a preset image to the calibration plate, and controlling a camera to collect an image of the calibration plate; performing joint optimization based on the re-projection error and the phase consistency constraint error determined by the calibration plate image to obtain a dual-target calibration error; performing joint optimization based on a left monocular system calibration error, a right monocular system calibration error, the dual-target calibration error, the monocular and binocular parameter consistency error and the three-dimensional point cloud consistency error to obtain a target calibration error; and performing calibration processing based on the target calibration error.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer, more particularly, relates to a projector-assisted binocular vision system calibration method, a binocular vision system calibration device, a binocular vision system calibration equipment and a computer readable storage medium. BACKGROUND

[0002] Structured light measurement is based on actively projecting a specific light pattern, capturing the deformed pattern on the surface of an object after modulation, and calculating the three-dimensional coordinate information of the object by combining triangulation. Compared with traditional contact measurement methods, structured light measurement technology has the advantages of high efficiency, simplicity and non-contact acquisition of three-dimensional information of the measured object, and has been widely used in industrial measurement and other fields.

[0003] In related technologies, for large-surface workpiece measurement, due to the influence of low calibration accuracy, insufficient calibration range, and close calibration data, the traditional binocular structured light system cannot guarantee both precision and measurement range, and there is a situation that the measurement precision is high in the central region and low in the edge region, resulting in a small available high-precision area. At the same time, the reconstruction area is usually the intersection of the common area of the binocular camera field of view and the structured light projection area, resulting in a small reconstruction area, which seriously affects the measurement efficiency of large-surface workpieces.

[0004] Therefore, how to expand the high-precision area after binocular vision system calibration and avoid affecting the measurement efficiency of large-surface workpieces is an important problem for those skilled in the art. SUMMARY

[0005] The purpose of the present application is to provide a projector-assisted binocular vision system calibration method, a binocular vision system calibration device, a binocular vision system calibration equipment and a computer readable storage medium to expand the high-precision area after binocular vision system calibration and improve the measurement efficiency of large-surface workpieces.

[0006] In view of the above defects or improvement needs of the prior art, the present application provides a projector-assisted binocular vision system calibration method, comprising:

[0007] Pre-calibrate the single-binocular structured light system based on Zhang's calibration method to obtain the internal and external parameters of the left single structured light system, the internal and external parameters of the right single structured light system, and the internal and external parameters of the binocular structured light system; wherein the binocular structured light system comprises a left camera, a right camera and a projector;

[0008] The inner and outer parameters of the left monocular structured light system, the inner and outer parameters of the right monocular structured light system, and the inner and outer parameters of the binocular structured light system are used to respectively perform calibration space calculation, so as to obtain a calibration space of the left monocular structured light system, a calibration space of the right monocular structured light system, and a calibration space of the binocular structured light system, and a target frame is calculated based on the size of the calibration board and each calibration space; wherein the target frame is used to guide the calibration board to be placed by a projector;

[0009] When the calibration board is placed, a projector is controlled to project a preset image on the calibration board, and a camera is controlled to capture a calibration board image; wherein the preset image includes a full-brightness image, a horizontal coded fringe image, and a vertical coded fringe image;

[0010] Joint optimization is performed based on the reprojection error and the phase consistency constraint error determined based on the calibration board image, so as to obtain a double-target calibration error;

[0011] Joint optimization is performed based on the left monocular system calibration error, the right monocular system calibration error, the double-target calibration error, a monocular-binar parameter consistency error, and a three-dimensional point cloud consistency error, so as to obtain a target calibration error;

[0012] Calibration processing is performed based on the target calibration error.

[0013] Optionally, the monocular-binar structured light system is pre-calibrated based on the Zhang calibration method, so as to obtain the inner and outer parameters of the left monocular structured light system, the inner and outer parameters of the right monocular structured light system, and the inner and outer parameters of the binocular structured light system, including:

[0014] The left camera and the right camera both capture a pre-calibration image;

[0015] Zhang calibration processing is performed based on the captured pre-calibration image, so as to obtain the inner and outer parameters of the left monocular structured light system, the inner and outer parameters of the right monocular structured light system, and the inner and outer parameters of the binocular structured light system.

[0016] Optionally, the inner and outer parameters of the left monocular structured light system, the inner and outer parameters of the right monocular structured light system, and the inner and outer parameters of the binocular structured light system are used to respectively perform calibration space calculation, so as to obtain a calibration space of the left monocular structured light system, a calibration space of the right monocular structured light system, and a calibration space of the binocular structured light system, and a target frame is calculated based on the size of the calibration board and each calibration space, including:

[0017] The inner and outer parameters of the left monocular structured light system, the inner and outer parameters of the right monocular structured light system, and the inner and outer parameters of the binocular structured light system are used to respectively perform calibration space calculation, so as to obtain a calibration space of the left monocular structured light system, a calibration space of the right monocular structured light system, and a calibration space of the binocular structured light system;

[0018] regionally partitioning each of the calibration spaces based on the size of the calibration board, to obtain a corresponding calibration board placement pose;

[0019] calculating corresponding pixel coordinates of four vertices of the calibration board in a camera pixel coordinate system and a projector pixel coordinate system based on each of the calibration board placement poses;

[0020] connecting the four vertex pixel coordinates of the calibration board to obtain the target frame.

[0021] Optionally, the calibration board is placed by guiding the calibration board through the projector, including:

[0022] The projector projects the target frame, and the camera image displays the target frame as an aid to guide the placement of the calibration board.

[0023] Optionally, the double-objective calibration error is obtained by jointly optimizing the re-projection error and the phase consistency constraint error determined based on the calibration board image, including:

[0024] The calibration board corner points are extracted based on the full-bright image of the calibration board image, and the re-projection error is obtained by constructing an error on the calibration board corner points through Zhang's calibration method;

[0025] The lateral absolute phase and the longitudinal absolute phase are obtained by phase unwrapping on images other than the full-bright image in the calibration board image through the phase shift method;

[0026] The phase consistency constraint error is obtained by constructing an error based on the phase difference between the same-named corner points in the calibration board corner points, the lateral absolute phase, and the longitudinal absolute phase;

[0027] The double-objective calibration error is obtained by minimizing and optimizing the first preset weight, the re-projection error, and the phase consistency constraint error through the Levenberg-Marquardt algorithm.

[0028] Optionally, the target calibration error is obtained by jointly optimizing the left monocular system calibration error, the right monocular system calibration error, the double-objective calibration error, the monocular-stereo parameter consistency error, and the three-dimensional point cloud consistency error, including:

[0029] The left monocular system calibration error and the right monocular system calibration error are obtained by constructing an error based on the corresponding monocular re-projection error and the corresponding monocular phase error;

[0030] The monocular-stereo parameter consistency error is obtained by constructing an error based on the internal and external parameters of the monocular structured light system and the internal and external parameters of the binocular structured light system;

[0031] The three-dimensional point coordinates of the binocular overlap area between the monocular structured light system and the binocular structured light system are consistent, error construction processing is performed, and a three-dimensional point cloud consistency error is obtained;

[0032] The second preset weight, the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular-binar parameter consistency error, and the three-dimensional point cloud consistency error are minimized and optimized by a Levenberg-Marquardt algorithm, and the target calibration error is obtained.

[0033] The application also provides a projector-assisted binocular vision system calibration device, comprising:

[0034] A pre-calibration module is configured to pre-calibrate a monocular-binar structured light system based on Zhang's calibration method, and obtain internal and external parameters of a left monocular structured light system, internal and external parameters of a right monocular structured light system, and internal and external parameters of a binocular structured light system; wherein the binocular structured light system comprises a left camera, a right camera, and a projector.

[0035] A calibration board placement module is configured to respectively perform calibration space calculation based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, and obtain a calibration space of the left monocular structured light system, a calibration space of the right monocular structured light system, and a calibration space of the binocular structured light system, and calculate a target frame based on the size of a calibration board and each of the calibration spaces; wherein the target frame is used to guide the calibration board to be placed by the projector.

[0036] An image acquisition module is configured to control the projector to project a preset image onto the calibration board after the calibration board is placed, and control the camera to acquire a calibration board image; wherein the preset image comprises a full-bright image, a horizontal coded fringe image, and a vertical coded fringe image.

[0037] A monocular-binar joint optimization module is configured to perform joint optimization based on a reprojection error and a phase consistency constraint error determined based on the calibration board image, and obtain a binocular calibration error.

[0038] A target optimization module is configured to perform joint optimization based on a left monocular system calibration error, a right monocular system calibration error, the binocular calibration error, a monocular-binar parameter consistency error, and a three-dimensional point cloud consistency error, and obtain a target calibration error.

[0039] A calibration processing module is configured to perform calibration processing based on the target calibration error.

[0040] Optionally, the calibration board placement module is specifically configured to: perform calibration space calculation based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, respectively, to obtain a calibration space of the left monocular structured light system, a calibration space of the right monocular structured light system, and a calibration space of the binocular structured light system; perform regional division on each of the calibration spaces based on the size of the calibration board, to obtain a corresponding calibration board placement pose; calculate corresponding pixel coordinates of four vertexes of the calibration board in a camera pixel coordinate system and a projector pixel coordinate system based on each of the calibration board placement poses; and connect the pixel coordinates of the four vertexes of the calibration board to obtain the target frame.

[0041] The application further provides a binocular vision system calibration device, comprising:

[0042] a memory for storing a computer program;

[0043] a processor for executing the computer program to implement the steps of the binocular vision system calibration method.

[0044] The application further provides a computer readable storage medium having a computer program stored thereon, wherein the computer program is executed by a processor to implement the steps of the binocular vision system calibration method.

[0045] The application provides a projector-assisted binocular vision system calibration method, which comprises the following steps: pre-calibration of a single-binocular structured light system based on Zhang's calibration method, to obtain internal and external parameters of a left single-binocular structured light system, internal and external parameters of a right single-binocular structured light system, and internal and external parameters of a binocular structured light system; wherein the binocular structured light system comprises a left camera, a right camera and a projector; calibration space calculation based on the internal and external parameters of the left single-binocular structured light system, the internal and external parameters of the right single-binocular structured light system and the internal and external parameters of the binocular structured light system, to obtain a calibration space of the left single-binocular structured light system, a calibration space of the right single-binocular structured light system and a calibration space of the binocular structured light system, and a target frame calculated based on the size of a calibration board and each calibration space; wherein the target frame is used to guide the placement of the calibration board by the projector; after the placement of the calibration board is completed, a preset image is projected onto the calibration board by the projector, and a camera is controlled to collect a calibration board image; wherein the preset image comprises a full-brightness image, a horizontal coded fringe image and a vertical coded fringe image; joint optimization based on the re-projection error and the phase consistency constraint error determined based on the calibration board image, to obtain a double-target calibration error; joint optimization based on the left single-binocular system calibration error, the right single-binocular system calibration error, the double-target calibration error, a single-binocular parameter consistency error and a three-dimensional point cloud consistency error, to obtain a target calibration error; and calibration processing based on the target calibration error.

[0046] The initial parameters are obtained by pre-calibration based on Zhang's calibration method, the calibration board is uniformly covered by combining calibration space calculation and target frame guidance, the re-projection error and the phase error are jointly optimized by using full-brightness and coded images, and the single-binocular and double-binocular multiple errors are fused for joint optimization, so that the full-view calibration accuracy is consistent. The method expands the high-precision area, solves the problems of low edge accuracy and small reconstruction area, significantly improves the measurement efficiency of large-scale surface workpieces, and provides high-precision basic data for industrial measurement. BRIEF DESCRIPTION OF DRAWINGS

[0047] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor based on the provided drawings.

[0048] Figure 1 A flowchart of a projector-assisted binocular vision system calibration method provided by the embodiments of the present application;

[0049] Figure 2 A single-binocular structured light system schematic diagram provided by the embodiments of the present application;

[0050] Figure 3 An initial pre-calibration flowchart provided by an embodiment of the present application;

[0051] Figure 4 A structured light system guide calibration flowchart provided by an embodiment of the present application;

[0052] Figure 5 A structured light system guide calibration schematic diagram provided by an embodiment of the present application;

[0053] Figure 6 A double-target calibration precision optimization schematic diagram using phase information provided by an embodiment of the present application;

[0054] Figure 7 A double-target calibration precision optimization flowchart using phase information provided by an embodiment of the present application;

[0055] Figure 8 A single-double eye joint optimization framework flowchart provided by an embodiment of the present application;

[0056] Figure 9 A structural schematic diagram of a projector-assisted binocular vision system calibration device provided by an embodiment of the present application;

[0057] Figure 10 A structural schematic diagram of a binocular vision system calibration device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] The purpose of the present application is to provide a projector-assisted binocular vision system calibration method, binocular vision system calibration device, binocular vision system calibration equipment, and computer readable storage medium, so as to expand the high-precision area after binocular vision system calibration and improve the measurement efficiency of large-area workpieces.

[0059] In order to make the purpose, technical solutions and advantages of the present application clearer and more apparent, the present application will be further described in detail below in combination with the drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and do not limit the present application. In addition, the technical features involved in each embodiment of the present application described below can be combined with each other as long as they do not conflict with each other.

[0060] The following describes a projector-assisted binocular vision system calibration method provided by the present application through an embodiment.

[0061] Reference is made to Figure 1 , Figure 1 A flowchart of a projector-assisted binocular vision system calibration method provided by an embodiment of the present application.

[0062] In this embodiment, the method can include:

[0063] S101, pre-calibrate the single-double monocular structured light system based on Zhang's calibration method to obtain the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system; wherein the binocular structured light system comprises a left camera, a right camera, and a projector;

[0064] The Zhang's calibration method in this step is a classic camera calibration method, which can solve the internal and external parameters of the camera by shooting images of calibration boards at different poses. In the present application, the left monocular structured light system (left camera + projector), the right monocular structured light system (right camera + projector), and the binocular structured light system (left camera + right camera + projector) are pre-calibrated to obtain the initial internal and external parameters, which provide basic data for subsequent accurate calibration. This is because the initial parameters are the premise of subsequent spatial calculation and error optimization. Only after the preliminary system model is determined, can more precise adjustment be made on this basis.

[0065] In this step, the Zhang's calibration method is used for pre-calibration, which provides initial parameters for the entire calibration process, establishes a preliminary model of the monocular and binocular systems, and provides basic data for subsequent calibration space division, target frame calculation, and error optimization, ensuring that the subsequent steps can be accurately adjusted and optimized on this basis.

[0066] Further, this step can include:

[0067] Step 1: The left camera and the right camera both collect pre-calibration images.

[0068] Step 2: Perform Zhang's calibration processing based on the collected pre-calibration images to obtain the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system.

[0069] S102, based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, respectively, perform calibration space calculation to obtain the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system, and calculate the target frame based on the size of the calibration board and each calibration space; wherein the target frame is used to guide the placement of the calibration board by the projector;

[0070] On the basis of S101, the calibration space (i.e. the field of view of each system) of the left monocular, right monocular and binocular structured light systems is calculated. Combined with the size of the calibration plate, each calibration space is divided into regions, and the placement posture (including position and posture) of the calibration plate is planned so that the calibration plate can evenly cover the entire calibration space. Then, the coordinates of the four vertices of the calibration plate in the camera pixel coordinate system and the projector pixel coordinate system are calculated for each posture, and these coordinates are connected to form a target frame. The principle is to ensure that the calibration data is evenly distributed in the entire measurement space by reasonably dividing the area and planning the posture, avoiding the problem of insufficient data in the edge area in the traditional method; the target frame serves as a visual guidance tool to help the operator accurately place the calibration plate to ensure that the calibration plate is in the preset optimal posture.

[0071] It can be seen that this step ensures that the calibration plate can be accurately placed according to the planned posture through calibration space calculation and target frame guidance, so that the calibration data evenly covers the entire measurement space, avoiding the problem of calibration data being concentrated in the central area and missing in the edge area, and laying the foundation for subsequent improvement of the calibration accuracy consistency of the entire measurement space.

[0072] Furthermore, this step may include:

[0073] Step 1: Based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system are calculated respectively;

[0074] Step 2: Divide each calibration space into regions based on the size of the calibration plate to obtain the corresponding calibration plate placement posture;

[0075] Step 3: Calculate the corresponding pixel coordinates of the four vertices of the calibration plate in the camera pixel coordinate system and the projector pixel coordinate system based on the placement posture of each calibration plate;

[0076] Step 4: Connect the pixel coordinates of the four vertices of the calibration plate to obtain the target frame.

[0077] Guiding the placement of the calibration plate by using a projector may include: the projector projects a target frame, and the camera image displays the target frame as an auxiliary to guide the placement of the calibration plate.

[0078] S103, after the calibration plate is placed, controlling the projector to project a preset image onto the calibration plate, and controlling the camera to capture an image of the calibration plate; wherein the preset image includes a full-brightness image, a horizontal coded stripe image, and a vertical coded stripe image;

[0079] On the basis of S102, when the calibration board is placed in place according to the target frame, the projector projects a set of preset images, including one full-bright image and 48 encoded fringe images (24 horizontal and 24 vertical). The full-bright image is used to extract the corner points of the calibration board, and the encoded fringe image is used to calculate the phase by the phase shift method to obtain absolute phase information. The camera synchronously captures these images to provide data support for subsequent error calculation. The full-bright image is used for corner point detection, and the encoded fringe image carries spatial position information through phase change. The combination of the two can simultaneously obtain feature point coordinates and phase information, providing multi-dimensional data for joint optimization.

[0080] In this step, the corner point coordinates and phase information of the calibration board are simultaneously obtained by capturing the full-bright image and the encoded fringe image, providing rich data for subsequent reprojection error calculation (based on the corner points of the calibration board) and phase consistency constraint error calculation (based on phase information), ensuring that the error optimization process can comprehensively utilize multiple information to improve the calibration accuracy.

[0081] S104, joint optimization of the reprojection error and the phase consistency constraint error determined based on the calibration board image to obtain a dual-target calibration error;

[0082] On the basis of S103, this step uses the corner points extracted from the full-bright image to calculate the reprojection error (i.e., the deviation of the actual corner point pixel coordinates from the model predicted coordinates) by Zhang's calibration method. Then, the horizontal and vertical absolute phases obtained from the encoded fringe image are used to construct the phase consistency constraint error by using the constraint that the phase values of the same name corner points of the left and right cameras should be equal. The two kinds of errors are combined and minimized by the Levenberg-Marquardt algorithm to obtain the dual-target calibration error. The principle is that the reprojection error reflects the geometric accuracy of the camera model, and the phase consistency error reflects the phase matching accuracy of the structured light system. Joint optimization of the two can improve the consistency of the camera parameters and the structured light parameters, avoiding the one-sidedness of relying on only geometric information or phase information.

[0083] In this step, the reprojection error and the phase consistency constraint error are jointly optimized to fully utilize the complementarity of geometric features and phase features, significantly improving the calibration accuracy of the binocular structured light system, especially in the edge area, where the phase constraint compensates for the possible deficiency of geometric features, making the internal and external parameters of the binocular camera more accurate.

[0084] Further, this step can include:

[0085] Step 1, extract the corner points of the calibration board based on the full-bright image of the calibration board image, and construct the error of the corner points of the calibration board by Zhang's calibration method to obtain the reprojection error;

[0086] Step 2, phase unwrapping the images in the calibration board image except the full-bright image by phase shifting method to obtain the transverse absolute phase and the longitudinal absolute phase;

[0087] Step 3, error construction based on the phase difference between the same named corner points in the calibration board corner points, transverse absolute phase and longitudinal absolute phase to obtain the phase consistency constraint error;

[0088] Step 4, minimizing and optimizing the first preset weight, the re-projection error and the phase consistency constraint error by Levenberg-Marquardt algorithm to obtain the dual target calibration error.

[0089] S105, joint optimization based on the left monocular system calibration error, the right monocular system calibration error, the dual target calibration error, the monocular and dual monocular parameter consistency error and the three-dimensional point cloud consistency error to obtain the target calibration error;

[0090] On the basis of S104, this step constructs a joint optimization framework, including the left monocular system calibration error (combined with monocular re-projection error and phase error), the right monocular system calibration error, the dual target calibration error, the monocular and dual monocular parameter consistency constraint error (ensuring the consistency of camera / projection parameters of monocular and dual monocular systems) and the three-dimensional point cloud consistency constraint error (forcing the consistency of three-dimensional point coordinates of monocular and dual monocular reconstruction in the dual monocular overlapping area). Weighted minimization of these errors by Levenberg-Marquardt algorithm obtains the target calibration error. The principle is that the monocular system works independently in the non-overlapping area, and the dual monocular system works in the overlapping area. Through joint optimization, the parameter difference and reconstruction error between monocular and dual monocular systems can be eliminated, and the calibration accuracy consistency of the entire measurement space (including monocular independent area and dual monocular overlapping area) is ensured.

[0091] It can be seen that this step optimizes multiple errors of monocular and dual monocular systems through joint optimization, not only improves the calibration accuracy of monocular system in independent area, but also ensures the consistency of parameters and reconstruction results of monocular and dual monocular systems in overlapping area, significantly reduces the accuracy difference between different areas, realizes high-precision calibration in full field of view, and expands the effective measurement range of the system.

[0092] Further, this step can include:

[0093] Step 1, error construction based on corresponding monocular re-projection error and corresponding monocular phase error to obtain left monocular system calibration error and right monocular system calibration error;

[0094] Step 2, error construction based on the internal and external parameters of monocular structured light system and the internal and external parameters of dual monocular structured light system to obtain monocular and dual monocular parameter consistency error;

[0095] Step 3, the three-dimensional point coordinates of the binocular overlap area between the monocular structured light system and the binocular structured light system are consistent, error construction processing is performed, and a three-dimensional point cloud consistency error is obtained;

[0096] Step 4, the second preset weight, the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular and binocular parameter consistency error, and the three-dimensional point cloud consistency error are minimized and optimized by the Levenberg-Marquardt algorithm, and a target calibration error is obtained.

[0097] S106, calibration processing is performed based on the target calibration error.

[0098] On the basis of S105, the target calibration error has been minimized through the foregoing joint optimization, at this time, final calibration processing is performed based on the optimized parameters, and accurate monocular and binocular structured light system internal and external parameters are obtained. These parameters can accurately describe the geometric relationship between the camera and the projector and the imaging model, and provide high-precision basic data for subsequent three-dimensional reconstruction and measurement.

[0099] In summary, the embodiment obtains initial parameters based on Zhang's calibration method, realizes uniform coverage of the calibration board by combining calibration space calculation and target frame guidance, uses full-bright and coded image joint optimization to re-project and phase error, and then fuses monocular and binocular multi-error for joint optimization to ensure consistent full-field calibration accuracy. This method expands the high-precision area, solves the problems of low edge accuracy and small reconstruction area, significantly improves the measurement efficiency of large-area workpieces, and provides high-precision basic data for industrial measurement.

[0100] The following further describes a projection-assisted binocular vision system calibration method provided by the application through another specific embodiment.

[0101] In this embodiment, the method can include:

[0102] S201, a monocular and binocular structured light vision system is built, and Zhang's calibration method is used to pre-calibrate the internal and external parameters of the left monocular system, the right monocular system, and the binocular system;

[0103] S202, the internal and external parameters of the monocular system and the binocular system are used to independently calculate the calibration spaces of the left monocular system, the right monocular system, and the binocular system, and the calibration spaces are regionally divided in combination with the size of the calibration board to plan the placement pose of the calibration board; wherein for the pose of each calibration board, the corresponding pixel coordinates of the four vertices of the calibration board in the camera pixel coordinate system and the projector pixel coordinate system are calculated, the pixel coordinates corresponding to the four vertices form a target frame, the projector projects the target frame, and the camera image of the calibration software displays the target frame, which is used to guide the placement of the calibration board;

[0104] S203, after the calibration board is placed to the corresponding pose area, the projector projects a group of images, including full-bright image and horizontal and vertical coded fringe image onto the calibration board, and the camera collects the calibration board image;

[0105] S204, in the double-target calibration calculation, the full-bright image in the collected image is subjected to calibration board corner point extraction, Zhang's calibration method is used to calculate the re-projection error, the coded fringe image is subjected to phase solution to obtain absolute phase, the phase values of the same-named calibration board corner points in the left and right camera images are equal, and a joint optimization equation of re-projection error and phase consistency error is constructed by using the constraint, and the double-target calibration precision is optimized;

[0106] S205, the double-target calibration result with higher precision is used as a constraint to construct a single-double-target calibration joint optimization equation, and the single-target calibration precision and the consistency of the single-double-target calibration precision are optimized and improved.

[0107] Please refer to Figure 2 , Figure 2 A single-double-target structured light system schematic diagram provided by the embodiment of the application.

[0108] As shown in the single-double-target structured light system of the embodiment of the application in Figure 2 , the single-double-target structured light system comprises a left camera C l , a projector P j and a right camera C r , the projection range of the projector P j is greater than the sum of the field of view of the left camera C l and the field of view of the right camera C r . The left camera C l and the projector P j form a left single-target structured light system S l , the right camera C r and the projector P j form a right single-target structured light system S r , and the left camera C l , the right camera C r and the projector P j jointly form a double-target structured light system S b .

[0109] In the field of view area division, area 1 is the field of view area of the left single-target structured light system S l , area 5 is the field of view area of the right single-target structured light system S r , area 3 is the field of view area of the double-target structured light system S b , which is also the field of view overlap area of the left single-target structured light system S l and the right single-target structured light system S r , and area 2 is the field of view area of the left single-target structured light system S lIndependent field of view region, i.e. with right monocular structured light system S r Non-overlapping field of view region, region 4 is right monocular structured light system S r Independent field of view region, i.e. with left monocular structured light system S l Non-overlapping field of view region.

[0110] Further, the calibration method workflow in the embodiment can also include:

[0111] Initial pre-calibration is performed before formal calibration, specifically: the calibration board is placed in region 3, the projector P j 49 images are projected in each group, of which the first image M0 is a full-brightness image, the second to twenty-fifth images M1, M2, …, M 24 are transverse coding stripes, and the twenty-sixth to forty-ninth images M 25 , M 26 , M 27 , …, M 48 are longitudinal coding stripes.

[0112] Please refer to Figure 3 , Figure 3 for an initial pre-calibration flowchart provided by the embodiment of the present application.

[0113] As Figure 3 , for the first three groups of images collected, the calibration parameters of the binocular structured light system S b are calculated by Zhang's calibration method using the full-brightness M0 image, and the calibration parameters of the left monocular structured light system S 48 and the calibration parameters of the right monocular structured light system S l are calculated by an improved method based on Zhang's calibration method using the full-brightness M0 and coding stripe images M1-M r . The calibration parameters include the intrinsic matrix K l of the left camera C l , the intrinsic matrix K r of the right camera C r , distortion coefficients (k1, k2), and extrinsic parameters (rotation matrix R, translation vector T). The measurement range in the depth direction [Z min , Z max ] is determined according to the actual measurement distance.

[0114] Please refer to Figure 4 , Figure 4 for a structured light system guided calibration flowchart provided by the embodiment of the present application

[0115] As Figure 4 , first determine the calibration space of the left and right monocular systems and the binocular system, divide the calibration space into regions and plan the placement pose of the calibration board, specifically as follows:

[0116] Step 1, the determination method of the calibration space is as follows: the left camera C l The field of view range 1 is:

[0117]

[0118] wherein, is the left camera C l horizontal and vertical field of view angles, is the left camera C work when the camera working distance is Z l the X-direction range of the field of view, is the left camera C work when the camera working distance is Z l the Y-direction range of the field of view, Z work is a distance value in the measurement range [Z min ,Z max ], representing the distance from the camera photosensitive element to the measured object. Similarly, in the system coordinate system with the left camera C l as the origin, the right camera C r The field of view range 5 is:

[0119]

[0120] wherein, b is the baseline distance, is the right camera C r horizontal and vertical field of view angles, is the right camera C work when the camera working distance is Z r the X-direction range of the field of view, is the right camera C work when the camera working distance is Z r the Y-direction range of the field of view, Z work is a distance value in the measurement range [Z min ,Z max ]. Through the field of view range 1 and the field of view range 5, the binocular overlapping area and the monocular non-overlapping area range can be obtained, wherein the left monocular non-overlapping field of view area 2 is:

[0121]

[0122] Similarly, the right monocular non-overlapping field of view area 4 is:

[0123]

[0124] The binocular overlapping field of view area 3 is:

[0125]

[0126] Step 2, the partition of the calibration region is as follows: first, in order to avoid the gap between the left monocular non-overlapping field of view region 2, the right monocular non-overlapping field of view region 4 and the binocular overlapping region 3, an expansion coefficient γ is introduced to expand the left monocular non-overlapping field of view region 2 and the right monocular non-overlapping field of view region 4:

[0127]

[0128] wherein w is the width of the calibration board, and the height of the calibration board is h. In order to align the region boundary of the calibration board at the head and tail positions, taking the X direction as an example, when the center of each calibration board moves in the X direction, the left and right edges thereof need to satisfy:

[0129]

[0130] wherein X start is the left end point of the region, X end is the right end point of the region, N x is the number of grid division in the X direction, is the center point of the calibration board at the first position in the X direction, is the center point of the calibration board at the tail position in the X direction. The interval L x of the center of the calibration board at the head and tail positions is:

[0131] L x = X end -X start -w

[0132] The step length Δx and the overlap coefficient α x in the X direction are calculated as:

[0133]

[0134] Similarly, the step length Δy and the overlap coefficient α y in the Y direction are calculated as:

[0135]

[0136] Step 3, the posture planning of the calibration board is as follows: for each calibration region, the calibration board rotates by an angle θ x , θ y , θ z about the X, Y and Z axes, while satisfying that the vertex coordinates P corner are within the calibration space:

[0137] P corner ∈[X min ,X max ]×[Y min ,Y max ]×[Z min ,Z max ]

[0138] The effective width w of the calibration board after rotation eff and the effective height h eff are:

[0139]

[0140] The above step and overlap coefficient are corrected using the effective width w eff and the effective height h eff For each set of calibration board rotation angles (θ x , θ y , θ z ), the appropriate grid division number N x and N y are selected to make the overlap coefficient α x , α y ∈(0,1) be at an appropriate value, satisfying the calibration efficiency and calibration accuracy requirements.

[0141] Please refer to Figure 5 , Figure 5 is a schematic diagram of the structured light system guided calibration provided by the embodiment of the application.

[0142] As shown in Figure 5 , for the left monocular non-overlapping area 2, only the left camera C l and the projector P j are guided, for the right monocular non-overlapping area 4, only the right camera C r and the projector P j are guided, and for the binocular overlapping area 3, the left camera C l , the right camera C r , and the projector P j are jointly guided.

[0143] For the placement order of the calibration board poses, whether the next placement pose of the calibration board is the optimal pose is calculated by an optimization algorithm starting from a calibration board pose, and the specific process is as follows: the optimal pose of the calibration board should maximize the accuracy of parameter estimation, which is usually measured by the Fisher Information Matrix (FIM). The larger the determinant of the FIM (D-optimality criterion), the smaller the covariance of parameter estimation, and the higher the accuracy.

[0144] The specific steps can include:

[0145] Step 1, calculate the current information matrix: for the set of calibration board poses {T k}, the FIM is:

[0146]

[0147] wherein, J kiis the Jacobian matrix of the re-projection error of the i-th point in the k-th pose with respect to the parameters.

[0148] Step 2, evaluate the amount of information: if det(I) exceeds a preset threshold or satisfies the local maximum condition, it is considered that the current pose is sufficient for optimization. Traverse all the planned poses of the calibration board in the binocular overlapping area 3, and select the best pose for the next placement.

[0149] Further, the positions of the four vertices of the calibration board in the left camera C l pixel coordinate system, the projector P j pixel coordinate system and the right camera C r pixel coordinate system in the best pose are calculated, specifically as follows:

[0150] Step 1, generate the four vertices P1, P2, P3, P4 of the calibration board in the best pose, and calculate the corresponding points P j , P p1 , P p2 , P p3 , P p4 in the projector P l pixel coordinate system, the corresponding points P cl1 , P cl2 , P cl3 , P cl4 in the left camera C r pixel coordinate system and the corresponding points P cr1 , P cr2 , P cr3 , P cr4 in the right camera C l pixel coordinate system. The coordinates of the points in the left camera C l pixel coordinate system are calculated as follows:

[0151] Taking the three-dimensional coordinates P l (X c , Y c , Z c ) of an angle point of the calibration board in the left camera C c coordinate system as an example, the conversion from the three-dimensional point to the image coordinate is calculated. The relationship between the point in the camera coordinate system and the point (x, y) in the image coordinate system is:

[0152]

[0153] where f is the focal length of the camera. The relationship between the point (x, y) in the image coordinate system and the point (u, v) in the pixel coordinate system is:

[0154]

[0155] Where dx and dy are the physical sizes of the pixel in the X and Y directions, respectively, and (u0, v0) is the position of the origin of the image coordinate system in the pixel coordinate system.

[0156] Combining the above equations, we can get the three-dimensional point P c (X c ,Y c ,Z c ) corresponds to the left camera C l The pixel coordinates of the image (u l ,v l )for:

[0157]

[0158]

[0159] Step 2, right camera C r The coordinates in the pixel coordinate system are calculated as follows:

[0160] It is known that the right camera C is calculated by the first three sets of images r With left camera C l The baseline distance and the right camera C r The internal and external parameter matrices of the right camera C r Point P in the coordinate system cr (X cr ,Y cr ,Z cr ) can be viewed through the right camera C r The external parameter matrix from the left camera C l The three-dimensional coordinate P in the coordinate system c (X c ,Y c ,Z c ) is converted to:

[0161]

[0162] Among them, R and T are the right camera C r Relative to the left camera C l The rotation matrix and translation vector.

[0163] Right camera C r Point P in the coordinate system cr (X cr ,Y cr ,Z cr ) is converted to the right camera C r The point (x r ,y r ) is:

[0164]

[0165] Right camera C r The point (x r ,y r ) is converted to a point in the pixel coordinate system (u r ,v r ) is:

[0166]

[0167] Among them, dx r and dy r Right camera C r The physical size of the pixel in the X and Y directions, (u 0r ,v 0r ) is the right camera C r The position of the image coordinate system origin in the pixel coordinate system.

[0168] Combining the above equations, we can get the three-dimensional point P cr (X cr ,Y cr ,Z cr ) corresponds to the right camera C r The pixels of the image (u r ,v r ) coordinates are:

[0169]

[0170] Step 3, Projector P j The coordinates in the pixel coordinate system are calculated as follows:

[0171] It is known that the projector P is calculated by the first three sets of images j The transformation matrix between the coordinate system and the left-phase coordinate system and the projector P j The internal and external parameter matrices of the projector P j Point P in the coordinate system p (X p ,Y p ,Z p ) can be achieved through the projector P j The external parameter matrix from the left camera C l Coordinate system conversion:

[0172]

[0173] Among them, R p and T p It is a projector P j Relative to the left camera C l The rotation and translation vectors of .

[0174] Projector Pj Point P in the coordinate system p (X p ,Y p ,Z p ) is converted into point (x p ,y p ) in the image coordinate system, the relationship is:

[0175]

[0176] Where f p is the focal length of the projector P j .

[0177] Point (x p ,y p ) in the image coordinate system of the projector P j is converted into point (u p ,v p ) in the pixel coordinate system, the relationship is:

[0178]

[0179] Where dx p and dy p are the physical sizes of the pixels of the projector P j in the X and Y directions, respectively, and (u 0p ,v 0p ) is the position of the origin of the coordinate system of the projector P j in the pixel coordinate system.

[0180] According to the above equations, the pixel coordinates (u 0p ,v 0p ) of the projector P j image corresponding to the three-dimensional point P p (X p ,Y p ,Z p ) can be obtained as:

[0181]

[0182] Step 4, when the top vertex of the calibration board is located in the binocular overlapping area 3, the projector P j projects a target frame composed of pixel points P p1 , P p2 , P p3 , and P p4 , and the left camera C l image in the calibration software displays a target frame composed of coordinate points P cl1 , P cl2 , P cl3 , and P cl4 , and the right camera C rThe image display consists of a target frame composed of coordinate points P cr1 , cr2 , cr3 , cr4 The target frame is used to guide the placement of the calibration board.

[0183] When the calibration board vertex is located in the left monocular non-overlapping area 2, the right camera C r has no corresponding point in the image, so it is guided by the left camera C l and the projector P j . Similarly, when the calibration board vertex is located in the right monocular non-overlapping area 4, it is guided by the left camera C r and the projector P j .

[0184] After the calibration board is placed in the specified pose, a set of images for calibration is collected, and the calibration image sequence is the same as in the pre-calibration process.

[0185] Please refer to Figure 6 , Figure 6 the schematic diagram for optimizing the precision of double target calibration provided by the embodiments of the present application using phase information.

[0186] Please refer to Figure 7 , Figure 7 the flowchart for optimizing the precision of double target calibration provided by the embodiments of the present application using phase information.

[0187] In order to ensure that the precision of double target calibration is high enough, phase consistency constraints are introduced in the double target calibration optimization, as follows:

[0188] Step 1: Extract the corner points of the calibration board using the full-bright M0 image in each group of images, calculate the re-projection error by Zhang's calibration method, and construct the re-projection error constraint:

[0189]

[0190] Where π l ,π r are the projection functions of the left camera C l and the right camera C r , including intrinsic parameters, extrinsic parameters and distortion models.

[0191] Step 2: Use the 2nd-49th images M1-M 48 of each group of images to solve the relative phase by the phase shift method and expand to get the horizontal absolute phase and the vertical absolute phase. For a corner point P l (u l ,v l ) in the image of the left camera C l , its horizontal phase value is Φ lx (u l ,vl ), the longitudinal phase value is Φ ly (u l ,v l ), the same-named corner point in the right camera C r image is found by corner point matching as P r (u r ,v r ), the transverse phase value is Φ rx (u r ,v r ), the longitudinal phase value is Φ ry (u r ,v r ), the phase consistency constraint error is constructed by using the phase difference of the same-named corner points:

[0192]

[0193] wherein, E phase-x and E phase-y represent the transverse phase and longitudinal phase consistency constraint errors respectively.

[0194] The dual-target optimization objective function is constructed by combining the re-projection error and the phase consistency error:

[0195] E stereo = λ1E phase-x + λ2E phase-y + λ3E reproj

[0196] wherein, λ1, λ2, λ3 are weight coefficients, used to balance the influence of the phase consistency constraint and the re-projection constraint in the optimization process. The optimization variables include camera intrinsic parameters (focal length f, principal point (u0, v0), distortion coefficients k1, k2), extrinsic parameters (rotation matrix R, translation vector T) and corner point coordinates, which are minimized by Levenberg-Marquardt algorithm. stereo In the optimization process, the transverse phase constraint and the longitudinal phase constraint jointly act, so that the same-named corner points of the left and right cameras have better phase consistency in the transverse and longitudinal directions, thereby improving the calibration accuracy of the binocular structured light system S b .

[0197] Further, in order to fuse the calibration errors of the left monocular structured light system S l , the right monocular structured light system S r and the binocular structured light system S b , a joint optimization framework is proposed, which reduces the single-target calibration and binocular calibration parameter difference and three-dimensional point cloud consistency error by introducing single-target calibration error, binocular calibration error, single-binoocular parameter consistency constraint and three-dimensional point cloud consistency constraint and performing joint optimization.

[0198] Please refer to Figure 8 , Figure 8 This is a flowchart of the monocular and binocular joint optimization framework provided in the embodiments of the present application.

[0199] To clearly describe the joint optimization framework, the following symbols and definitions are added:

[0200] Monocular projection model:

[0201] Left camera C l +ProjectorP j Model: π lp (·)(Left Camera C l Parameter Θ1+projector P j Parameter Θ p ).

[0202] Right camera C r +ProjectorP j Model: π rp (·)(Right Camera C r Parameter Θ r +ProjectorP j Parameter Θ p ).

[0203] 3D point consistency region:

[0204] Corner point X of the calibration plate observed by both binocular and monocular i ,The coordinates reconstructed by monocular and binocular are consistent.

[0205] Parameter sharing constraints:

[0206] Camera parameters Θ in binocular positioning l ,Θ r Keep the same parameters as those in single target calibration.

[0207] like Figure 8 , the errors of each part are described as follows:

[0208] Single target positioning error:

[0209] For the monocular structured light system, the monocular reprojection error (the projection error between the camera observation coordinates and the monocular reconstruction points) and the monocular phase error (the projector P j The single-target positioning error equation is constructed based on the phase coordinates and the projection phase consistency of the monocular reconstruction point.

[0210] Left camera C l The calibration error of the monocular structured light system is:

[0211]

[0212] Right camera C rThe calibration error of the monocular structured light system is:

[0213]

[0214] wherein and are three-dimensional points independently reconstructed by the monocular system.

[0215] The calibration error of the binocular system is:

[0216] Based on the foregoing binocular calibration error, there are:

[0217]

[0218] The monocular-binoocular parameter consistency constraint is:

[0219] The binocular calibration and the monocular calibration are forced to share the same set of camera and projector parameters, avoiding parameter inconsistency, and the parameter consistency error equation is:

[0220]

[0221] The three-dimensional point cloud consistency constraint is:

[0222] In the binocular and monocular common observation part, i.e., the binocular overlapping area 3, the monocular and binocular reconstructed three-dimensional point coordinates are forced to be consistent, and the three-dimensional point coordinate consistency error is:

[0223]

[0224] The joint optimization equation is:

[0225] The foregoing four items are combined with weights to form a total objective function:

[0226] E total = λ1E mono-l + λ2E mono-r + λ3E param + λ4E point + λ5E stereo

[0227] E total is minimized by the Levenberg-Marquardt optimization algorithm, the calibration accuracy of the monocular and binocular systems is improved, and the difference between the monocular calibration parameters and the binocular calibration parameters and the three-dimensional point cloud consistency error are reduced.

[0228] The embodiment actively guides the pose of the calibration board in the calibration process by projecting a guide frame by the projector and a marked target frame in the camera image of the calibration software, without introducing a special device to simplify the calibration conditions, and avoiding the need for recalibration due to changes in the position of the structured light system or the position of the calibration device during the calibration process.

[0229] The embodiment guarantees that the calibration region of the calibration plate covers the entire measurement space while the calibration plate is in an optimal pose by dynamic guidance of calibration space division and calibration plate pose, thereby significantly improving the calibration accuracy consistency of the entire measurement space of the structured light system.

[0230] The embodiment projects a full-bright image and an encoded image on the calibration plate by the projector, and simultaneously acquires the calibration plate image and the structured light encoded image, which are respectively used for feature point detection and decoding to obtain phase information. By using the phase consistency of the left and right homonymous points as a constraint, a joint optimization equation of re-projection error and phase consistency error is constructed, which can significantly improve the calibration accuracy of the binocular camera.

[0231] The embodiment jointly optimizes the single-target calibration error and the dual-target calibration error, which can significantly improve the single-target calibration accuracy and reduce the difference between the single-target calibration error and the dual-target calibration error. The introduction of the single-dual camera parameter consistency constraint can significantly reduce the difference between the single-target calibration parameters and the dual-target calibration parameters, thereby ensuring the consistency of the single-dual target calibration parameters. The introduction of the three-dimensional point cloud consistency constraint forces the three-dimensional point coordinates reconstructed by the monocular and binocular cameras to be consistent, thereby significantly improving the point cloud consistency of the fusion area of the monocular reconstruction and the binocular reconstruction. Finally, the construction of the single-dual camera joint optimization equation can ensure the consistency of the single-target calibration accuracy and the dual-target calibration accuracy, the consistency of the monocular reconstruction accuracy and the binocular reconstruction accuracy, and the fusion degree of the reconstructed point cloud in the large-view structured light system.

[0232] A projection-assisted binocular vision system calibration device provided by an embodiment of the application will be described below. The projection-assisted binocular vision system calibration device described below can be correspondingly referred to the projection-assisted binocular vision system calibration method described above.

[0233] Please refer to Figure 9 , Figure 9 FIG. 1 is a structural schematic diagram of a projection-assisted binocular vision system calibration device provided by an embodiment of the application.

[0234] In the embodiment, the device can include:

[0235] A pre-calibration module 100 is configured to pre-calibrate the single-dual structured light system based on Zhang's calibration method to obtain the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system. The binocular structured light system includes a left camera, a right camera, and a projector.

[0236] The calibration board placing module 200 is configured to perform calibration space calculation based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, to obtain the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system, and to calculate a target frame based on the size of the calibration board and each calibration space; the target frame is used to guide the calibration board to be placed by the projector.

[0237] The image acquisition module 300 is configured to control the projector to project a preset image onto the calibration board after the calibration board is placed, and to control the camera to acquire an image of the calibration board; the preset image includes a full-brightness image, a horizontal coded fringe image, and a vertical coded fringe image.

[0238] The monocular-binoocular joint optimization module 400 is configured to perform joint optimization based on the reprojection error and the phase consistency constraint error determined based on the image of the calibration board, to obtain a binocular calibration error.

[0239] The target optimization module 500 is configured to perform joint optimization based on the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular-binoocular parameter consistency error, and the three-dimensional point cloud consistency error, to obtain a target calibration error.

[0240] The calibration processing module 600 is configured to perform calibration processing based on the target calibration error.

[0241] Optionally, the calibration board placing module is specifically configured to perform calibration space calculation based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, to obtain the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system; to perform regional division on each calibration space based on the size of the calibration board, to obtain a corresponding calibration board placing pose; to calculate corresponding pixel coordinates of four vertexes of the calibration board in the camera pixel coordinate system and the projector pixel coordinate system based on each calibration board placing pose; and to connect the pixel coordinates of the four vertexes of the calibration board to obtain a target frame.

[0242] The application also provides a binocular vision system calibration device, please refer to Figure 10 , Figure 10 The binocular vision system calibration device provided by the application can include:

[0243] The memory is configured to store the computer program;

[0244] The processor is configured to perform the steps of the above-mentioned any one kind of projector assisted binocular vision system calibration method when executing the computer program.

[0245] like Figure 10 FIG2 is a schematic diagram showing the structure of a binocular vision system calibration device, which may include a processor 10, a memory 11, a communication interface 12, and a communication bus 13. The processor 10, the memory 11, and the communication interface 12 communicate with each other via the communication bus 13.

[0246] In the embodiment of the present application, the processor 10 may be a central processing unit (CPU), an application-specific integrated circuit, a digital signal processor, a field programmable gate array, or other programmable logic devices.

[0247] The processor 10 may call a program stored in the memory 11 . Specifically, the processor 10 may execute the operations in the embodiment of the abnormal IP identification method.

[0248] The memory 11 is used to store one or more programs. The program may include program code, and the program code includes computer operating instructions. In the embodiment of the present application, the memory 11 stores at least a program for implementing the following functions:

[0249] The monocular and binocular structured light systems are pre-calibrated based on Zhang's calibration method to obtain the internal and external parameters of the left monocular structured light system, the right monocular structured light system, and the binocular structured light system. The binocular structured light system includes a left camera, a right camera, and a projector.

[0250] Based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system are respectively calculated. The target frame is calculated based on the size of the calibration plate and each calibration space; wherein, the target frame is used to guide the placement of the calibration plate through the projector;

[0251] After the calibration plate is placed, the projector is controlled to project a preset image onto the calibration plate, and the camera is controlled to capture the image of the calibration plate; wherein the preset image includes a full-brightness image, a horizontal coded stripe image, and a vertical coded stripe image;

[0252] The reprojection error and phase consistency constraint error determined based on the calibration plate image are jointly optimized to obtain the dual-target positioning error.

[0253] The target calibration error is obtained by joint optimization based on the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular and binocular parameter consistency error, and the 3D point cloud consistency error.

[0254] The calibration process is performed based on the target calibration error.

[0255] In a possible implementation, the memory 11 can include a program storage area and a data storage area, where the program storage area can store an operating system and at least one application required by a function, etc., and the data storage area can store data created during use.

[0256] In addition, the memory 11 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device or other volatile solid-state memory device.

[0257] The communication interface 12 can be an interface of a communication module, used for connecting with other devices or systems.

[0258] Of course, it needs to be explained that, Figure 10 The structure shown does not constitute a limitation on the binocular vision system calibration device in the embodiments of the present application, and in actual application, the binocular vision system calibration device can include more or fewer components than Figure 10 those shown, or combine certain components.

[0259] The present application also provides a computer readable storage medium, which stores a computer program, and the computer program is executed by a processor to implement the steps of any one of the above projection-assisted binocular vision system calibration methods.

[0260] The computer readable storage medium can include a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and various media that can store program codes.

[0261] For the computer readable storage medium provided by the present application, refer to the above method embodiments, and the present application will not be repeated here.

[0262] The embodiments in the specification are described in a progressive manner, and each embodiment focuses on the difference from other embodiments. The same or similar parts of each embodiment can be referred to each other. For the device disclosed by the embodiments, since it corresponds to the method disclosed by the embodiments, the description is relatively simple, and the related parts can be referred to the method part.

[0263] Those skilled in the art will further appreciate that the functionality of the various examples illustrated in the figures can be implemented together or separately as part of a larger system. Accordingly, for example, a component such as a module can be implemented and / or deployed, including a component such as an algorithm, a process running to execute an algorithm, an object, an individual, and / or a thread.

[0264] Steps of methods or algorithms described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in random access memory (RAM), non-volatile memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0265] The above provides a kind of projector assisted binocular vision system calibration method, binocular vision system calibration device, binocular vision system calibration equipment, computer readable storage medium provided in the application are introduced in detail.In this paper, the principle and implementation of the present application are described by applying specific examples, and the above example is only used to help understand the method and its core idea of the present application.It should be pointed out that for ordinary skilled in the art, without departing from the principle of the present application, the present application can be improved and modified in several ways, and these improvements and modifications also fall within the protection scope of the claims of the present application.

Claims

1. A projector-assisted binocular vision system calibration method, characterized in that: include: Pre-calibrate the monocular and binocular structured light systems based on Zhang's calibration method to obtain the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system; wherein the binocular structured light system includes: a left camera, a right camera, and a projector; Calibration spaces are calculated based on internal and external parameters of the left monocular structured light system, internal and external parameters of the right monocular structured light system, and internal and external parameters of the binocular structured light system to obtain the calibration spaces of the left monocular structured light system, the right monocular structured light system, and the binocular structured light system, and a target frame is calculated based on the size of a calibration plate and each of the calibration spaces; wherein the target frame is used to guide the placement of the calibration plate via a projector; After the calibration plate is placed, the projector is controlled to project a preset image onto the calibration plate, and the camera is controlled to capture the calibration plate image; wherein the preset image includes a full-brightness image, a horizontal coded stripe image, and a vertical coded stripe image; performing joint optimization based on the reprojection error and the phase consistency constraint error determined based on the calibration plate image to obtain a dual-target calibration error; The target calibration error is obtained by performing joint optimization based on the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular and binocular parameter consistency error, and the three-dimensional point cloud consistency error; A calibration process is performed based on the target calibration error.

2. The binocular vision system calibration method according to claim 1, characterized in that: The monocular and binocular structured light systems are pre-calibrated based on Zhang's calibration method to obtain the internal and external parameters of the left monocular structured light system, the right monocular structured light system, and the binocular structured light system, including: The left camera and the right camera both capture pre-calibrated images; Zhang calibration processing is performed based on the collected pre-calibrated image to obtain the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system.

3. The binocular vision system calibration method according to claim 2, characterized in that: Calibration spaces are calculated based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system to obtain the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system, and a target frame is calculated based on the size of the calibration plate and each of the calibration spaces, including: performing calibration space calculations based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, respectively, to obtain the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system; Dividing each calibration space into regions based on the size of the calibration plate to obtain a corresponding calibration plate placement posture; Calculating the corresponding pixel coordinates of the four vertices of the calibration plate in the camera pixel coordinate system and the projector pixel coordinate system based on the placement posture of each calibration plate; The target frame is obtained by connecting the pixel coordinates of the four vertices of the calibration plate.

4. The binocular vision system calibration method according to claim 3, characterized in that: The calibration plate is placed by guiding it through a projector, including: The projector projects the target frame, and the camera image displays the target frame as an aid to guide the placement of the calibration plate.

5. The binocular vision system calibration method according to claim 4, characterized in that: The reprojection error and the phase consistency constraint error determined based on the calibration plate image are jointly optimized to obtain a dual-target calibration error, including: Extracting corner points of the calibration plate based on the full-brightness image of the calibration plate image, performing error construction on the corner points of the calibration plate by Zhang's calibration method, and obtaining a reprojection error; Performing phase unwrapping on the images except the full-brightness image in the calibration plate image by a phase shift method to obtain a horizontal absolute phase and a vertical absolute phase; Perform error construction based on the corner points of the calibration plate, the horizontal absolute phase, and the phase difference between the corner points with the same name in the vertical absolute phase to obtain a phase consistency constraint error; The first preset weight, the reprojection error, and the phase consistency constraint error are minimized and optimized using the Levenberg-Marquardt algorithm to obtain the dual-target positioning error.

6. The binocular vision system calibration method according to claim 5, characterized in that: The target calibration error is obtained by performing joint optimization based on the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular and binocular parameter consistency error, and the 3D point cloud consistency error, including: Performing error construction processing based on the corresponding monocular reprojection error and the corresponding monocular phase error to obtain the left monocular system calibration error and the right monocular system calibration error; Performing error construction processing based on the internal and external parameters of the monocular structured light system and the internal and external parameters of the binocular structured light system to obtain the monocular and binocular parameter consistency error; The three-dimensional point coordinates of the binocular overlapping area between the monocular structured light system and the binocular structured light system are aligned, and error construction processing is performed to obtain a three-dimensional point cloud consistency error; The second preset weight, the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular and binocular parameter consistency error, and the three-dimensional point cloud consistency error are minimized and optimized by the Levenberg-Marquardt algorithm to obtain the target calibration error.

7. A projector-assisted binocular vision system calibration device, characterized in that: include: A pre-calibration module is used to pre-calibrate the monocular and binocular structured light systems based on Zhang's calibration method to obtain internal and external parameters of the left monocular structured light system, the right monocular structured light system, and the binocular structured light system; wherein the binocular structured light system includes: a left camera, a right camera, and a projector; A calibration plate placement module is configured to calculate calibration spaces based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, respectively, to obtain the calibration spaces of the left monocular structured light system, the right monocular structured light system, and the binocular structured light system, and to calculate a target frame based on the size of the calibration plate and each of the calibration spaces; wherein the target frame is used to guide the placement of the calibration plate via a projector; An image acquisition module is used to control the projector to project a preset image onto the calibration plate after the calibration plate is placed, and to control the camera to acquire the calibration plate image; wherein the preset image includes a full-brightness image, a horizontal coded stripe image, and a vertical coded stripe image; A monocular and binocular joint optimization module is used to perform joint optimization based on the reprojection error and the phase consistency constraint error determined based on the calibration plate image to obtain a binocular positioning error; A target optimization module is used to perform joint optimization based on the left monocular system calibration error, the right monocular system calibration error, the binocular calibration error, the monocular and binocular parameter consistency error, and the three-dimensional point cloud consistency error to obtain a target calibration error; A calibration processing module is used to perform calibration processing based on the target calibration error.

8. The binocular vision system calibration device according to claim 7, characterized in that: The calibration plate placement module is specifically used to perform calibration space calculation based on the internal and external parameters of the left monocular structured light system, the internal and external parameters of the right monocular structured light system, and the internal and external parameters of the binocular structured light system, respectively, to obtain the calibration space of the left monocular structured light system, the calibration space of the right monocular structured light system, and the calibration space of the binocular structured light system; Dividing each calibration space into regions based on the size of the calibration plate to obtain a corresponding calibration plate placement posture; Calculating the corresponding pixel coordinates of the four vertices of the calibration plate in the camera pixel coordinate system and the projector pixel coordinate system based on the placement posture of each calibration plate; The target frame is obtained by connecting the pixel coordinates of the four vertices of the calibration plate.

9. A binocular vision system calibration device, characterized in that: include: Memory for storing computer programs; A processor, configured to implement the steps of the binocular vision system calibration method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps of the binocular vision system calibration method according to any one of claims 1 to 7 are implemented.

Images