Calibration method and device for multi-path depth camera system

Abstract

The invention provides a calibration method and device for a multi-path depth camera system, and the method comprises the steps: obtaining a multi-surface three-dimensional calibration plate with a configuration adaptive to the layout of cameras and a known geometric structure, placing the multi-surface three-dimensional calibration plate in a system view field, and guaranteeing that each camera at least observes one complete plane of the multi-surface three-dimensional calibration plate; all the cameras are controlled to shoot once, and infrared and depth images with aligned pixels are obtained; for each camera, extracting two-dimensional coordinates of feature points from an infrared image of the camera, and querying a corresponding depth value to construct an initial three-dimensional point set; and performing depth correction on the initial point set based on the known geometric structure of the calibration plate. According to the invention, through the specially designed calibration plate and the single shooting process, rapid and high-precision large field angle calibration is realized, and the operation threshold is greatly reduced.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and 3D reconstruction technology, and in particular to a calibration method and apparatus for a multi-channel depth camera system. Background Technology

[0002] Depth cameras, capable of directly acquiring 3D depth information of a scene, have been widely used in 3D modeling, industrial measurement, mixed reality, and human recognition. However, the field of view of a single depth camera is usually limited, making it impossible to capture a 360-degree view of an object or scene in one shot. Furthermore, the measurement accuracy of depth cameras decreases with increasing distance; to ensure high accuracy, the shooting distance is often limited to a relatively short range, further shrinking the effective shooting area.

[0003] To overcome the aforementioned limitations, applications requiring high-precision, wide-range (e.g., 360-degree) imaging typically employ array systems composed of multiple depth cameras. For this system to function correctly, the intrinsic parameter matrix (describing the camera's own optical characteristics) and the extrinsic parameter matrix (describing the spatial position and orientation relationships between cameras) of each camera must be accurately obtained; this process is known as calibration.

[0004] Existing calibration techniques have many shortcomings. In multi-camera systems, especially those used for 360-degree surround shooting, there is often a lack of shared visual areas between cameras, posing a significant challenge to extrinsic parameter calibration. Traditional calibration methods typically require operators to hold a planar calibration board (such as a checkerboard) and take dozens of images from different positions and postures within the camera's field of view to provide sufficient constraints for solving for camera parameters. This process is not only complex and time-consuming but also demands a high level of expertise from the operator, and the quality of the calibration results heavily depends on the operator's experience.

[0005] Some existing technologies attempt to address this problem. For example, some solutions propose using 3D targets, but to ensure accuracy, additional high-precision measurement equipment such as LiDAR is still required as a reference, which not only increases costs but also does not lower the operational threshold. Other solutions attempt to directly utilize depth data for registration, but due to the high noise in depth images and the blurring of edge and corner information, it is difficult to stably and accurately extract feature points for calibration, resulting in low final calibration accuracy that cannot meet the needs of high-precision applications. Summary of the Invention

[0006] This invention provides a calibration method and apparatus for a multi-channel depth camera system, which solves the defects of existing multi-channel depth camera system calibration processes that are complex, time-consuming, technically demanding, and difficult to handle cameras without shared viewing areas. It enables rapid and convenient calibration of intrinsic and extrinsic parameters of multi-channel, wide-field-of-view, and non-shared-view depth camera systems.

[0007] This invention provides a calibration method for a multi-channel depth camera system, comprising: A stereo calibration plate with multiple planes is obtained and placed within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein, the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern. The system controls the multi-channel depth camera system to take a single shot to acquire image data from each depth camera. The image data includes an infrared image and a depth image, and the infrared image and the depth image are pixel-aligned. For each depth camera, extract the two-dimensional image coordinates of the preset feature points on the observed calibration pattern from its infrared image, and query the corresponding depth value to construct an initial three-dimensional point set; Based on the known geometry of the stereo calibration plate, the initial three-dimensional point set is depth-corrected.

[0008] The calibration method for a multi-channel depth camera system provided by the present invention further includes: Based on the known geometry of the stereo calibration plate, a virtual 3D point set on a plane not observed by the current camera is generated. The corrected initial 3D point set is then merged with the virtual 3D point set to obtain a complete 3D point set. Based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set, the extrinsic parameter matrix between the multiple depth cameras is solved and output.

[0009] According to the calibration method for a multi-channel depth camera system provided by the present invention, the step of extracting the two-dimensional image coordinates of preset feature points on the observed calibration pattern from its infrared image and querying the corresponding depth values ​​to construct an initial three-dimensional point set specifically includes: Using the coordinates of the two-dimensional image as an index, the depth value is obtained from the depth image acquired in the same frame; By combining the intrinsic parameter model of the depth camera, the two-dimensional image coordinates of each feature point and the depth value are converted into three-dimensional spatial coordinates in the current camera coordinate system to form the initial three-dimensional point set.

[0010] According to the calibration method for a multi-channel depth camera system provided by the present invention, the step of depth correction of the initial three-dimensional point set based on the known geometry of the stereo calibration plate specifically includes: The points in the initial three-dimensional point set that correspond to the same plane are fitted to a plane, and these points are projected onto the fitted plane.

[0011] According to the calibration method for a multi-channel depth camera system provided by the present invention, the step of generating a virtual 3D point set on a plane not observed by the current camera based on the known geometry of the stereo calibration plate specifically includes: Determine the initial pose of the current camera relative to the stereo calibration plate; Based on the initial pose and the known geometry of the stereo calibration plate, the unobserved feature points defined in the global coordinate system of the stereo calibration plate are transformed to the coordinate system of the current camera to form the virtual 3D point set.

[0012] According to the calibration method for a multi-channel depth camera system provided by the present invention, before performing depth correction on the initial three-dimensional point set based on the known geometry of the stereo calibration plate, the method further includes: The initial 3D point set is processed using a random sampling consensus algorithm to remove outliers.

[0013] According to the calibration method of the multi-channel depth camera system provided by the present invention, the calibration pattern is a checkerboard pattern or a QR code pattern, and the feature points are the corner points of the checkerboard pattern or the corner points of the QR code; in the multi-channel depth camera system, at least two depth cameras do not have a common visible area.

[0014] According to the calibration method for a multi-channel depth camera system provided by the present invention, after solving and outputting the extrinsic parameter matrix between the multi-channel depth cameras, the method further includes: The reprojection error of the complete 3D point set is calculated based on the solved extrinsic matrix; When the error exceeds a preset threshold, the system prompts the user to adjust the position of the stereo calibration plate and re-execute the acquisition steps.

[0015] The present invention also provides a calibration device for a multi-channel depth camera system, comprising the following modules: The calibration plate acquisition module is used to acquire a stereo calibration plate with multiple planes and place the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein, the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern; The image acquisition module is used to control the multi-channel depth camera system to take a picture once to acquire image data from each depth camera. The image data includes infrared images and depth images, and the infrared images and depth images are pixel-aligned. The initial point set construction module is used to extract the two-dimensional image coordinates of preset feature points on the observed calibration pattern in the infrared image of each depth camera, and query the corresponding depth value to construct an initial three-dimensional point set. The complete point set generation module is used to perform depth correction on the initial three-dimensional point set based on the known geometric structure of the stereo calibration plate.

[0016] The calibration device for a multi-channel depth camera system provided by the present invention further includes a solution module; The complete point set generation module is further configured to generate a virtual three-dimensional point set on a plane not observed by the current camera based on the known geometric structure of the stereo calibration plate, and merge the corrected initial three-dimensional point set with the virtual three-dimensional point set to obtain a complete three-dimensional point set. The solution module is used to solve and output the extrinsic parameter matrix between the multiple depth cameras based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set. In the multi-channel depth camera system, at least two depth cameras do not share a common field of view.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the calibration method of any of the above-described multi-channel depth camera systems.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the calibration method for the multi-channel depth camera system as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the calibration method of any of the multi-channel depth camera systems described above.

[0020] The calibration method and apparatus for a multi-channel depth camera system provided by this invention involves acquiring a multifaceted stereo calibration plate with a configuration adapted to the camera layout and known geometry, and placing it within the system's field of view to ensure that each camera observes at least one complete plane. All cameras are controlled to take one shot, acquiring pixel-aligned infrared and depth images. For each camera, the two-dimensional coordinates of feature points are extracted from its infrared image, and the corresponding depth values ​​are queried to construct an initial three-dimensional point set. Based on the known geometry of the calibration plate, the initial point set is depth-corrected. This invention achieves fast, high-precision, large field-of-view calibration through a specially designed calibration plate and a single-shot process, significantly reducing the operational threshold. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is one of the flowcharts illustrating the calibration method for a multi-channel depth camera system provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of the multi-channel depth camera calibration system provided by the present invention.

[0024] Figure 3 This is the second flowchart illustrating the calibration method for a multi-channel depth camera system provided by the present invention.

[0025] Figure 4 This is a schematic diagram of the calibration device for the multi-channel depth camera system provided by the present invention.

[0026] Figure 5 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0028] In the description of the embodiments of the present invention, it should be noted that the terms "upper," "lower," "front," "rear," "left," and "right," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are only for the convenience of describing the embodiments of the present invention and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the embodiments of the present invention. In addition, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0029] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of the present invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.

[0030] In the description of this invention, unless otherwise stated, "at least one" includes one or more. "More than one" means two or more. For example, at least one of A, B, and C includes: A existing alone, B existing alone, A and B existing simultaneously, A and C existing simultaneously, B and C existing simultaneously, and A, B, and C existing simultaneously. In this invention, " / " means "or," for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0031] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. The specific operation methods in the method embodiments can also be applied to the device embodiments or system embodiments. Obviously, the described embodiments are some, but not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0032] In some specific embodiments of the present invention, such as Figure 1 As shown, this solution provides a calibration method for a multi-channel depth camera system, including: Step 100: Obtain a stereo calibration plate with multiple planes and place the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein, the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern; Step 200: Control the multi-channel depth camera system to take a picture once to obtain image data from each depth camera. The image data includes infrared images and depth images, and the infrared images and depth images are pixel-aligned. Step 300: For each depth camera, extract the two-dimensional image coordinates of the preset feature points on the observed calibration pattern in its infrared image, and query the corresponding depth value to construct an initial three-dimensional point set. Step 400: Based on the known geometry of the stereo calibration plate, perform depth correction on the initial three-dimensional point set.

[0033] In some possible embodiments of the present invention, the calibration method for a multi-channel depth camera system provided by the present invention further includes: Step 500: Based on the known geometry of the stereo calibration plate, generate a virtual 3D point set on the plane that the current camera has not observed, and merge the corrected initial 3D point set with the virtual 3D point set to obtain a complete 3D point set; Step 600: Based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set, solve and output the extrinsic parameter matrix between the multiple depth cameras.

[0034] It should be noted that existing calibration schemes for multi-channel depth cameras require the use of an additional LiDAR as a reference for calibration to ensure accuracy. Furthermore, the fused multi-channel depth cameras after calibration can only capture a range of less than 120 degrees. Not only is the calibration process technically demanding, but the field of view for capture is also relatively small. When directly using depth data for calibration, it is limited by the fact that depth data is less robust than infrared data, has higher noise levels, and makes it difficult to find accurate calibration corner points, resulting in lower calibration accuracy and failing to meet the high-precision calibration requirements of multi-channel depth imaging modules.

[0035] Specifically, in environments requiring high-precision, 360-degree imaging, multiple cameras need to work simultaneously, and the depth data needs to be fused with high precision. This requires calculating the intrinsic parameters of each camera and the extrinsic parameters between cameras. In most usage environments, there is no overlap between camera images, further increasing the difficulty of extrinsic parameter calibration between modules. Furthermore, to complete calibration, multiple calibration images need to be taken at different locations using a calibration board to prevent overdeterminism during the calculation process. Therefore, in actual calibration, existing technologies are complex, have high operational barriers, and are time-consuming. This makes it difficult for non-professionals to independently complete the calibration process for multi-depth camera devices, especially in high-precision applications where deformation of the device necessitates repeated calibration during use.

[0036] Therefore, this invention designs a high-precision multi-faceted stereo calibration plate. The structural precision of this plate addresses the need for additional high-precision measuring instruments for calibration. It satisfies calibration requirements exceeding 360 degrees of field of view, especially challenging calibrations where cameras have no common viewing area. Simultaneously, leveraging the principle of infrared and depth sharing a common source, and combining high-precision infrared corner detection with stereo information from the depth map, it achieves extrinsic and intrinsic parameter calibration for multiple depth cameras with a large field of view exceeding 360 degrees in a single calibration. On one hand, it solves the problem of calibrating and registering multiple depth cameras with a 360-degree field of view and no shared viewing area; on the other hand, it overcomes the challenge of traditional calibration methods that require multiple shots of the calibration plate at different locations and the assistance of high-precision imaging equipment. This significantly lowers the barrier to entry for calibration personnel.

[0037] The above steps will be explained in detail below through specific embodiments.

[0038] Step 100: Obtain a stereo calibration plate with multiple planes and place the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein, the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern; In some possible embodiments of the present invention, the calibration pattern is a checkerboard pattern or a QR code pattern, and the feature points are the corner points of the checkerboard pattern or the corner points of the QR code.

[0039] Specifically, this embodiment provides an implementation method for calibrating patterns and feature points, providing reliable and easily extractable features, such as checkerboard corner points and QR code corner points, which have very mature and high-precision sub-pixel level detection algorithms (such as the OpenCV library), ensuring the accuracy and stability of two-dimensional coordinate extraction.

[0040] In some possible embodiments of the present invention, in the multi-channel depth camera system, at least two depth cameras do not share a common field of view.

[0041] This embodiment is specifically designed to address the thorny problem of the lack of a common visible area, which is difficult to handle with traditional methods, and is distinctly different from existing technologies that rely on overlapping fields of view.

[0042] Step 200: Control the multi-channel depth camera system to take a picture once to obtain image data from each depth camera. The image data includes infrared images and depth images, and the infrared images and depth images are pixel-aligned. Step 300: For each depth camera, extract the two-dimensional image coordinates of the preset feature points on the observed calibration pattern in its infrared image, and query the corresponding depth value to construct an initial three-dimensional point set. In some possible embodiments of the present invention, the step of extracting the two-dimensional image coordinates of preset feature points on the observed calibration pattern from its infrared image and querying the corresponding depth values ​​to construct an initial three-dimensional point set specifically includes: Using the coordinates of the two-dimensional image as an index, the depth value is obtained from the depth image acquired in the same frame; By combining the intrinsic parameter model of the depth camera, the two-dimensional image coordinates of each feature point and the depth value are converted into three-dimensional spatial coordinates in the current camera coordinate system to form the initial three-dimensional point set.

[0043] Specifically, this embodiment provides an implementation method for constructing an initial 3D point set. In memory, the infrared image and the depth image can be regarded as two arrays with the same height and width. After reading the pixel coordinates (u, v) of the corner points in the infrared image, these coordinates are directly used as indices to access the same position (u, v) in the depth image array and read the stored floating-point number (depth value Z). Then, using the pre-stored camera intrinsic parameter matrix K, the 3D coordinates are calculated using the inverse perspective projection transformation formula. By utilizing the pixel alignment characteristics of the infrared image and the depth image, a fast and accurate 2D to 3D conversion is achieved, eliminating the complex calculation of finding corresponding points through matching and searching in the depth image, and avoiding matching errors caused by noise and missing textures. By directly utilizing the inherent geometric correspondence of the hardware, the data association is more reliable.

[0044] Step 400: Based on the known geometry of the stereo calibration plate, perform depth correction on the initial three-dimensional point set; In some possible embodiments of the present invention, before performing depth correction on the initial three-dimensional point set based on the known geometry of the stereo calibration plate, the method further includes: The initial 3D point set is processed using a random sampling consensus algorithm to remove outliers.

[0045] Specifically, this embodiment provides an implementation method for preprocessing an initial 3D point set. After obtaining the initial 3D point set, the RANSAC algorithm is invoked. This algorithm randomly samples the minimum point set (3 points for a plane) to fit a model, then calculates the distance from all points to the model and counts the number of interior points. After multiple iterations, the model with the most interior points is selected, and all interior points are retained while the remaining exterior points are discarded. This embodiment can effectively remove erroneous 3D points (exterior points) caused by noise in the depth map and mis-extraction of infrared map corner points (such as those falling on areas of pattern contamination), enhancing anti-interference capabilities.

[0046] In some possible embodiments of the present invention, the depth correction of the initial three-dimensional point set based on the known geometry of the stereo calibration plate specifically includes: The points in the initial three-dimensional point set that correspond to the same plane are fitted to a plane, and these points are projected onto the fitted plane.

[0047] Specifically, this embodiment provides an implementation method for depth correction of an initial 3D point set. The least squares method is used to fit multiple 3D points belonging to the same calibration plate plane into a single plane equation. Then, each original 3D point is vertically projected onto this fitted plane. The coordinates of the projected points are the corrected coordinates, which better conform to the actual physical plane. This embodiment constrains discrete, noisy 3D points to an ideal geometric plane, effectively smoothing measurement errors, especially depth jumps caused by distant or low-reflectivity areas, thereby suppressing depth noise. It ensures that all points on the same physical plane strictly satisfy the plane equation, providing more accurate and consistent geometric constraints for subsequent calculations and improving the consistency of calibration data.

[0048] Step 500: Based on the known geometry of the stereo calibration plate, generate a virtual 3D point set on the plane that the current camera has not observed, and merge the corrected initial 3D point set with the virtual 3D point set to obtain a complete 3D point set; In some possible embodiments of the present invention, the generation of a virtual 3D point set on a plane not observed by the current camera based on the known geometry of the stereo calibration plate specifically includes: Determine the initial pose of the current camera relative to the stereo calibration plate; Based on the initial pose and the known geometry of the stereo calibration plate, the unobserved feature points defined in the global coordinate system of the stereo calibration plate are transformed to the coordinate system of the current camera to form the virtual 3D point set.

[0049] Specifically, this embodiment provides an implementation method for constructing a virtual three-dimensional point set, which uses the global prior geometric model of the calibration board to supplement the data not observed by the camera, thereby achieving calibration without common viewing area and expanding the effective constraints.

[0050] In a possible embodiment, the pose of the current camera relative to the calibration board plane it observes can be solved using the observed points and algorithms such as PnP. Given a complete CAD model of the calibration plate (the coordinates of each corner point in the calibration plate's own coordinate system {B}). ). Combination Fixed transformation relationship between the planes of the calibration plate Calculate the coordinates of all corner points (including unobserved ones) of the entire calibration board in the current camera coordinate system: = These calculated points constitute a virtual point set.

[0051] With the above settings, even if the camera only sees a portion of the calibration board, it can still deduce the complete model's shape in its own coordinate system from the known structure. This provides a shared and complete spatial reference for different cameras, achieving calibration without common viewing areas. It significantly increases the number of 3D points used to solve for extrinsic parameters, and these points have strict geometric relationships with the observation points, forming stronger optimization constraints and making the solution more stable and accurate.

[0052] Step 600: Based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set, solve and output the extrinsic parameter matrix between the multiple depth cameras.

[0053] In some possible embodiments of the present invention, after solving and outputting the extrinsic parameter matrix between the multiple depth cameras, the method further includes: The reprojection error of the complete 3D point set is calculated based on the solved extrinsic matrix; When the error exceeds a preset threshold, the system prompts the user to adjust the position of the stereo calibration plate and re-execute the acquisition steps.

[0054] Specifically, this embodiment provides an implementation method for verifying the extrinsic parameter matrix between multiple depth cameras obtained by solving. The calibration accuracy is measured by the reprojection error, providing an objective assessment of the accuracy. When the accuracy is insufficient, the position is adjusted and data is collected again for high-quality calibration.

[0055] The following will refer to the appendices in the embodiments of the present invention. Figure 2-3 The technical solutions in the embodiments of the present invention will be described more clearly and completely. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0056] Please see Figure 2 This illustrates the structure of a multi-channel depth camera calibration system according to one embodiment of the present invention. The system includes a multi-channel depth camera system 100 and a multi-faceted stereo calibration plate 200. The system may also include a processing device.

[0057] The multi-channel depth camera system 100 includes at least two depth cameras, such as camera 110 in the figure. These cameras are mounted on a bracket to form a preset, fixed spatial layout, such as the wraparound layout shown in example 2, to achieve 360-degree shooting. Each depth camera is capable of outputting multiple image data, including at least an infrared image, a depth image, and a color image. Crucially, its output infrared and depth images are pixel-aligned, meaning that any pixel coordinate (u, v) in the infrared image corresponds physically to the same point in space as the pixel at the same coordinate (u, v) in the depth image.

[0058] The multi-faceted calibration plate 200 is one of the key components of this invention. It has multiple (e.g., six or twelve) planes, such as plane 210 as shown in the figure. The calibration plate 200 is manufactured using high-precision machining (e.g., CNC), and the three-dimensional geometric relationships, such as the angles and distances between its planes, are precisely known in advance. Each plane of the calibration plate 200 has a calibration pattern 220 for feature extraction, such as a checkerboard or QR code pattern. The configuration of the calibration plate 200 is specially designed to adapt to the preset spatial layout of the multi-channel depth camera system 100. Adaptation means that when the calibration plate 200 is placed at the center of the system's workspace, it ensures that each camera 110 can clearly and completely observe at least one plane of the calibration plate 200.

[0059] The processing device can be a personal computer (PC), an industrial control computer, or an embedded system, and its internal components include a processor, memory, input / output interfaces, etc. The memory stores a computer program, which the processor executes to implement the calibration method described in this invention. All cameras 110 are connected to the processing device via a data cable (such as USB) or a network.

[0060] The following is combined Figure 3 The flowchart shown illustrates in detail the calibration method of the present invention.

[0061] Step S301: Obtain and place the stereo calibration plate.

[0062] In this step, firstly, a multi-faceted stereo calibration plate 200 is obtained based on the actual layout of the multi-channel depth camera system 100 to be calibrated. Then, the calibration plate 200 is placed within the field of view of the system 100. The placement must meet one condition: each depth camera 110 in the system must be able to see at least one complete plane of the calibration plate 200 within its field of view. For example, in… Figure 2 In the diagram, the left camera 110 observes the left plane 210, and the right camera 110 observes the right plane 210. It is worth noting that this invention can handle situations where the cameras do not share a common viewing area, such as when the fields of view of the left and right cameras do not overlap at all.

[0063] Step S302: Perform a synchronized shot.

[0064] In this step, once the calibration board 200 is in place, the recording is triggered via the software interface on the processing device. The processing device controls all cameras 110 to perform a synchronized recording, requiring the calibration board 200 to remain stationary at the moment of recording. Through this recording, the processing device acquires image data from each camera, primarily infrared images and pixel-aligned depth images.

[0065] Step S303: Construct the initial 3D point set.

[0066] In this step, for each camera in the system (taking the left camera as an example): First, the processing device uses image processing algorithms (such as the findChessboardCorners function in OpenCV) to extract the two-dimensional image coordinates {(u,v)} of all preset feature points (such as chessboard corner points) on the observed calibration pattern 220 from the infrared image it has acquired.

[0067] Then, taking advantage of the pixel alignment between the infrared image and the depth image, for each extracted two-dimensional image coordinate (u,v), the depth value Z stored at the (u,v) position of the depth image in the same frame is directly queried using this as an index.

[0068] Finally, combining the intrinsic parameter matrix of the camera (which can be pre-calibrated or used as a variable to be optimized), the image information (u,v) and depth information Z of each feature point are converted into three-dimensional spatial coordinates (X, Y, Z) in the coordinate system of the camera 110a using the inverse operation of the camera projection model. The collection of the converted three-dimensional coordinates of all feature points on a plane constitutes the initial three-dimensional point set of the camera.

[0069] Step S304: Generate a complete 3D point set.

[0070] The purpose of this step is to optimize and expand the noisy initial 3D point set using the prior structural information of the calibration board.

[0071] In an optional embodiment, before performing this step, the initial 3D point set generated in step S303 can be preprocessed using the Random Sampling Consensus (RANSAC) algorithm to remove outliers that are obviously erroneous due to occlusion, reflection, etc., thereby improving the robustness of subsequent calculations.

[0072] This step specifically includes two sub-steps: Sub-step 1: Depth Correction. Due to inherent errors in depth camera measurements, the initial 3D point set constructed directly from the depth map is not entirely accurate. For example, points that should be on the same plane may be scattered in 3D space. Therefore, prior knowledge of the high flatness of the calibration plate plane is needed for correction. Specifically, for all 3D points in the initial 3D point set corresponding to the same observation plane, plane fitting (e.g., using the Singular Value Decomposition (SVD) algorithm) is performed to obtain a best-fit plane equation. Then, each point in this set of 3D points is orthogonally projected onto this best-fit plane. After this step, the coplanarity error of the point set is eliminated, and its accuracy is closer to the physical accuracy of the calibration plate, thus obtaining the corrected initial 3D point set.

[0073] Sub-step 2: Generate a virtual 3D point set. The left camera can only see the left plane, but the complete 3D model of the entire calibration board 200 is known. To provide stronger constraints for subsequent solutions, it is necessary to supplement the parts that the left camera does not see. The specific method is: First, using the corrected initial 3D point set (in the coordinate system of the left camera) obtained in the previous step and their corresponding 2D image coordinates, the initial pose of the left camera relative to the stereo calibration plate 200 (i.e., rotation matrix R and translation vector T) can be calculated by solving a PnP (Perspective-n-Point) problem.

[0074] Then, we have the precise 3D coordinates of all feature points on the calibration board in their own global coordinate system. Using the initial pose (R,T) we just solved, we can transform the feature points on the plane that are not directly observed by the left camera from the global coordinate system of the calibration board to the camera coordinate system of the left camera. These points obtained after coordinate transformation are the virtual 3D point set.

[0075] Finally, the corrected initial 3D point set is merged with the virtual 3D point set to obtain the complete 3D point set corresponding to the camera. This complete point set theoretically contains the precise 3D coordinates of all feature points on the calibration board in the current camera coordinate system.

[0076] Steps S303 and S304 are repeated for each camera in the system.

[0077] Step S305: Solve for the extrinsic parameter matrix.

[0078] After the above steps, for all cameras in the system, we have obtained a set of two-dimensional feature point image coordinates and a corresponding complete set of three-dimensional points (these three-dimensional points are all in the same global coordinate system, namely the calibration board).

[0079] At this point, the problem transforms into a global optimization problem. We construct a cost function that takes the extrinsic parameters (and optional intrinsic parameters) of all cameras as variables and the total error (i.e., reprojection error) between the reprojection positions of all observed 2D feature points and their corresponding 3D points on their respective images as the objective.

[0080] By minimizing the cost function using a nonlinear optimization algorithm, the poses of all cameras relative to the calibration board coordinate system can be calculated in one go with high precision. Since all cameras are aligned to the same global coordinate system, their extrinsic parameter matrices (i.e., their relative pose relationships) can be easily calculated.

[0081] In an optional embodiment, after solving for the extrinsic parameter matrix, a verification step can be added. This involves calculating the final reprojection error and determining if it is less than a preset accuracy threshold. If the error is too large, it indicates poor image quality (e.g., improper placement of the calibration board, obstruction, etc.). The processing device can then provide corresponding prompts, guiding the user to adjust the position of the calibration board and restart from step S302 to ensure the quality of the final calibration result.

[0082] The calibration method for a multi-channel depth camera system provided in this invention simplifies the complex process of repeatedly shooting dozens of times in different poses, as required by traditional methods, to a single successful shot by employing a specially designed multi-faceted stereo calibration plate. No professional experience is required from the operator; simply placing the calibration plate correctly is sufficient. This significantly shortens calibration time, reduces operational difficulty, greatly simplifies the calibration process, and lowers the barrier to entry. By having all cameras observe the same rigid stereo calibration plate, even if they observe different planes and have no overlapping fields of view, a unified coordinate system can be established through the calibration plate. This allows for accurate calculation of their relative pose relationships, effectively solving the calibration problem for cameras without shared viewing areas, which is particularly crucial for the construction of systems such as 360-degree surround shooting systems. Furthermore, this invention utilizes the characteristic that infrared and depth images from a depth camera are of the same origin and pixel-aligned, extracting corner points from the clearer, sharper infrared image, thus ensuring the accuracy of the two-dimensional coordinates. Simultaneously, it uses the high-precision physical structure of the calibration board itself as a benchmark to correct depth measurement errors and virtually generates unobserved points to enhance the data, thereby achieving higher calibration accuracy than directly using noisy depth data, improving calibration precision and robustness. This invention relies on the structural precision of the calibration board itself, eliminating the need for expensive high-precision auxiliary measurement equipment such as lidar, thereby reducing the overall system construction and maintenance costs and lowering system costs.

[0083] In some specific embodiments of the present invention, such as Figure 4 As shown, this solution provides a calibration device for a multi-channel depth camera system, comprising: The calibration plate acquisition module 10 is used to acquire a stereo calibration plate with multiple planes. The configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system. The geometry between each plane is known in advance, and each plane is provided with a calibration pattern. The stereo calibration plate is placed within the field of view of the system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate. The image acquisition module 20 is used to control the multi-channel depth camera system to take a picture once to acquire image data from each depth camera. The image data includes an infrared image and a depth image, and the infrared image and the depth image are pixel-aligned. The initial point set construction module 30 is used to extract the two-dimensional image coordinates of preset feature points on the observed calibration pattern in the infrared image of each depth camera, and query the corresponding depth value to construct an initial three-dimensional point set. The complete point set generation module 40 is used to perform depth correction on the initial three-dimensional point set based on the known geometric structure of the stereo calibration plate.

[0084] In some possible embodiments of the present invention, the calibration device of the multi-channel depth camera system further includes a solution module 50; The complete point set generation module is further configured to generate a virtual three-dimensional point set on a plane not observed by the current camera based on the known geometric structure of the stereo calibration plate, and merge the corrected initial three-dimensional point set with the virtual three-dimensional point set to obtain a complete three-dimensional point set. The solving module 50 is used to solve and output the extrinsic parameter matrix between the multiple depth cameras based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set.

[0085] These modules work together to complete the entire calibration process.

[0086] It is possible that in the multi-channel depth camera system, at least two depth cameras do not share a common field of view.

[0087] As can be seen from the above embodiments of the present invention, the present invention simplifies the complex multi-channel depth camera calibration problem into a simple task that can be completed in a single shot through a cleverly designed stereo calibration board and optimized algorithm process. It effectively solves the pain points of the prior art, such as complex calibration process and inability to handle areas without common view, and has extremely high practical value.

[0088] Figure 5 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 5As shown, the electronic device may include: a processor 510, a communications interface 520, a memory 530, and a communications bus 540, wherein the processor 510, the communications interface 520, and the memory 530 communicate with each other through the communications bus 540. The processor 510 can call logic instructions in the memory 530 to execute a calibration method for a multi-channel depth camera system. This method includes: acquiring a stereo calibration plate with multiple planes and placing the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein the configuration of the stereo calibration plate is adapted to a preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane has a calibration pattern; controlling the multi-channel depth camera system to take one shot to acquire image data from each depth camera, the image data including infrared images and depth images, and the infrared images and depth images are pixel-aligned; for each depth camera, extracting the two-dimensional image coordinates of preset feature points on the observed calibration pattern in its infrared image and querying the corresponding depth value to construct an initial three-dimensional point set; and performing depth correction on the initial three-dimensional point set based on the known geometry of the stereo calibration plate.

[0089] Furthermore, the logical instructions in the aforementioned memory 530 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0090] On the other hand, the present invention also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can execute the calibration method for a multi-channel depth camera system provided by the above methods. The method includes: acquiring a stereo calibration plate with multiple planes and placing the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometric structure between its planes is known in advance, and a calibration pattern is provided on each plane; controlling the multi-channel depth camera system to take a picture once to acquire image data of each depth camera, the image data including infrared images and depth images, and the infrared images and depth images are pixel-aligned; for each depth camera, extracting the two-dimensional image coordinates of preset feature points on the observed calibration pattern in its infrared image and querying the corresponding depth value to construct an initial three-dimensional point set; and performing depth correction on the initial three-dimensional point set based on the known geometric structure of the stereo calibration plate.

[0091] In another aspect, the present invention also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements a calibration method for a multi-channel depth camera system provided by the methods described above. The method includes: acquiring a stereo calibration plate having multiple planes and placing the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein the configuration of the stereo calibration plate is adapted to a preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern; controlling the multi-channel depth camera system to take a picture once to acquire image data of each depth camera, the image data including an infrared image and a depth image, and the infrared image and the depth image are pixel-aligned; for each depth camera, extracting the two-dimensional image coordinates of preset feature points on the observed calibration pattern in its infrared image and querying the corresponding depth value to construct an initial three-dimensional point set; and performing depth correction on the initial three-dimensional point set based on the known geometry of the stereo calibration plate.

[0092] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0093] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0094] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A calibration method for a multi-channel depth camera system, characterized in that, include: A stereo calibration plate with multiple planes is obtained and placed within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein, the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern. The system controls the multi-channel depth camera system to take a single shot to acquire image data from each depth camera. The image data includes an infrared image and a depth image, and the infrared image and the depth image are pixel-aligned. For each depth camera, extract the two-dimensional image coordinates of the preset feature points on the observed calibration pattern from its infrared image, and query the corresponding depth value to construct an initial three-dimensional point set; Based on the known geometry of the stereo calibration plate, the initial three-dimensional point set is depth-corrected.

2. The calibration method for a multi-channel depth camera system according to claim 1, characterized in that, Also includes: Based on the known geometry of the stereo calibration plate, a virtual 3D point set on a plane not observed by the current camera is generated. The corrected initial 3D point set is then merged with the virtual 3D point set to obtain a complete 3D point set. Based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set, the extrinsic parameter matrix between the multiple depth cameras is solved and output.

3. The calibration method for a multi-channel depth camera system according to claim 1, characterized in that, The process of extracting the two-dimensional image coordinates of preset feature points on the observed calibration pattern from its infrared image and querying the corresponding depth values ​​to construct an initial three-dimensional point set specifically includes: Using the coordinates of the two-dimensional image as an index, the depth value is obtained from the depth image acquired in the same frame; By combining the intrinsic parameter model of the depth camera, the two-dimensional image coordinates of each feature point and the depth value are converted into three-dimensional spatial coordinates in the current camera coordinate system to form the initial three-dimensional point set.

4. The calibration method for a multi-channel depth camera system according to claim 1, characterized in that, The depth correction of the initial 3D point set based on the known geometry of the 3D calibration plate specifically includes: The points in the initial three-dimensional point set that correspond to the same plane are fitted to a plane, and these points are projected onto the fitted plane.

5. The calibration method for a multi-channel depth camera system according to claim 2, characterized in that, The known geometric structure based on the stereo calibration plate generates a virtual 3D point set on a plane not observed by the current camera, specifically including: Determine the initial pose of the current camera relative to the stereo calibration plate; Based on the initial pose and the known geometry of the stereo calibration plate, the unobserved feature points defined in the global coordinate system of the stereo calibration plate are transformed to the coordinate system of the current camera to form the virtual 3D point set.

6. The calibration method for a multi-channel depth camera system according to claim 1, characterized in that, Before performing depth correction on the initial 3D point set based on the known geometry of the stereo calibration plate, the method further includes: The initial 3D point set is processed using a random sampling consensus algorithm to remove outliers.

7. The calibration method for a multi-channel depth camera system according to claim 1, characterized in that, The calibration pattern is a checkerboard pattern or a QR code pattern, and the feature points are the corner points of the checkerboard pattern or the corner points of the QR code; in the multi-channel depth camera system, at least two depth cameras do not have a common visible area.

8. The calibration method for a multi-channel depth camera system according to any one of claims 1-7, characterized in that, After solving and outputting the extrinsic parameter matrix between the multiple depth cameras, the method further includes: The reprojection error of the complete 3D point set is calculated based on the solved extrinsic matrix; When the error exceeds a preset threshold, the system prompts the user to adjust the position of the stereo calibration plate and re-execute the acquisition steps.

9. A calibration device for a multi-channel depth camera system, characterized in that, include: The calibration plate acquisition module is used to acquire a stereo calibration plate with multiple planes and place the stereo calibration plate within the field of view of the multi-channel depth camera system to ensure that any depth camera can observe at least one complete plane of the stereo calibration plate; wherein, the configuration of the stereo calibration plate is adapted to the preset spatial layout of the multi-channel depth camera system, the geometry between its planes is known in advance, and each plane is provided with a calibration pattern; The image acquisition module is used to control the multi-channel depth camera system to take a picture once to acquire image data from each depth camera. The image data includes infrared images and depth images, and the infrared images and depth images are pixel-aligned. The initial point set construction module is used to extract the two-dimensional image coordinates of preset feature points on the observed calibration pattern in the infrared image of each depth camera, and query the corresponding depth value to construct an initial three-dimensional point set. The complete point set generation module is used to perform depth correction on the initial three-dimensional point set based on the known geometric structure of the stereo calibration plate.

10. The calibration device for a multi-channel depth camera system according to claim 9, characterized in that, It also includes a solver module; The complete point set generation module is further configured to generate a virtual three-dimensional point set on a plane not observed by the current camera based on the known geometric structure of the stereo calibration plate, and merge the corrected initial three-dimensional point set with the virtual three-dimensional point set to obtain a complete three-dimensional point set. The solution module is used to solve and output the extrinsic parameter matrix between the multiple depth cameras based on the two-dimensional image coordinates of the feature points and the complete three-dimensional point set. In the multi-channel depth camera system, at least two depth cameras do not share a common field of view.

Images