Global calibration and fusion method and system for multi-camera 3D stitching and storage medium
By combining FPGA synchronous triggering and GPU parallel computing with hybrid feature matching and weighted fusion, the problems of poor synchronization, difficult calibration and low accuracy in multi-camera stitching are solved, realizing efficient and real-time multi-camera 3D stitching, which is suitable for large industrial scenarios.
Patent Information
- Application Number
- CN202611221128.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-12
- Publication Date
- 2026-11-17
AI Technical Summary
Existing multi-camera stitching technology suffers from problems such as poor synchronization, difficult calibration, low accuracy, and insufficient efficiency in large industrial scenarios. In particular, timing deviations lead to large stitching errors in the detection of moving workpieces, calibration parameters drift and lack online self-calibration, point cloud processing is time-consuming, fusion algorithms fail to match weak texture surfaces, and there is serious duplication of calculations.
By triggering nanosecond-level exposures of multiple 3D cameras with FPGA synchronization pulses, a global joint optimization objective function is constructed to synchronously solve for the temporal offset of external participants. Hybrid feature matching and weighted fusion are adopted, combined with GPU parallel computing and cache reuse, to achieve high-precision and fast global calibration and fusion.
It achieves micron-level accuracy and real-time performance for multi-camera stitching, solves the problems of poor synchronization, difficult calibration, and slow stitching, improves computing efficiency, and is suitable for high-end industrial scenarios such as large workpiece inspection and robot grasping.
Smart Images

Figure CN123118780A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of industrial 3D machine vision technology, specifically to a global calibration and fusion method, system, and storage medium for multi-camera 3D stitching. Background Technology
[0002] Currently, large-scale 3D reconstruction of industrial scenes generally employs an array of multiple 3D cameras, using multi-view point cloud stitching to achieve full-area coverage. However, existing technologies have technical limitations in actual production line environments, making it difficult to balance accuracy, speed, and stability.
[0003] At the data acquisition level, traditional multi-camera systems rely on software frame synchronization triggering, with exposure time differences between cameras typically ranging from 5ms to 20ms. For moving workpieces, this timing deviation directly introduces uncompensated spatial misalignment, easily causing stitching errors to exceed 1mm, which cannot meet the requirements for micron-level detection. At the global calibration level, conventional checkerboard or target methods require all cameras to have a common overlapping field of view. However, in large-span encircling layouts, adjacent cameras often have little or no overlap. Insufficient feature matching points lead to beam adjustment optimization outside the calibration process easily getting trapped in local optima, making it difficult to unify global coordinates. Furthermore, environmental temperature changes and mechanical creep cause calibration parameters to continuously drift, and the lack of an online self-calibration mechanism results in decreased long-term operational reliability.
[0004] At the point cloud processing level, current solutions perform serial iterative registration in pairs, with the time generally exceeding 80ms when the number of cameras exceeds four, severely restricting production line cycle time. When facing weak textures or highly reflective surfaces such as metals and plastics, simple geometric distance-based stitching and fusion algorithms fail, frequently producing holes, ghosting, and layering defects. Regarding fusion strategies, the depth confidence varies significantly among cameras due to differences in distance and angle. Existing algorithms treat each source point cloud with equal weight, causing low-confidence noise to spread globally, significantly reducing overall accuracy. Simultaneously, massive amounts of redundant data remain uncompressed, hindering subsequent computational efficiency. Furthermore, in the production of batches of workpieces of the same specifications, the entire calibration and stitching process is repeated each time, lacking cached reuse of calibration parameters and standard templates. Extensive redundant computation wastes computing power, making it difficult to match the pace of high-speed online inspection. Summary of the Invention
[0005] In view of this, this application provides a global calibration and fusion method, system and storage medium for multi-camera 3D stitching to solve the technical problems of poor synchronization, difficult calibration, low accuracy and insufficient efficiency in the prior art.
[0006] Firstly, this application provides a global calibration and fusion method for multi-camera 3D stitching, including: Multiple 3D cameras are triggered to expose synchronously with nanosecond precision using FPGA synchronization pulses. The depth point cloud and 2D texture images output by each camera are acquired in parallel, and distortion correction and invalid depth point removal are performed on the point cloud of each camera. Based on the collected cross-field feature points, a global joint optimization objective function containing the intrinsic parameters, extrinsic parameters, and temporal offsets of each camera is constructed and solved iteratively in a synchronous manner. Global coordinate transfer between non-overlapping fields of view is completed through intermediate transition cameras. The global transformation matrix of each camera relative to the world coordinate system is output. The extrinsic parameters are periodically corrected based on the real-time extracted workpiece geometric features to compensate for calibration drift. Using the global transformation matrix as the initial mapping, a hybrid descriptor of geometric and texture features is extracted synchronously for each point cloud. After removing mismatched point pairs through rigid transformation consistency check, the global space is divided into three-dimensional voxel blocks and allocated to different computing cores of the GPU to perform cross-camera feature coarse matching in parallel, thus completing the initial global alignment of multi-view point clouds. Using the coarsely registered aligned point cloud as input, the depth confidence weight of each 3D point is calculated based on the shooting distance and incident angle of each camera. A weighted point-to-plane distance objective function is constructed and a robust kernel function is embedded. The residual misalignment is eliminated by iterative reweighted least squares optimization. Overlapping areas are fused according to confidence level to retain high confidence points. Finally, an adaptive voxel block strategy is used to compress and output the global point cloud. The global calibration parameters and the complete point cloud template of the standard workpiece are stored in a high-speed cache. When processing a batch of workpieces of the same specification, the cached data is directly called. Only the minor deviations between the current frame and the template are lightly fine-tuned, and the cache is automatically updated based on the splicing error monitoring results. The fused global dense point cloud is uniformly converted to the world coordinate system, the workpiece's three-dimensional pose and size data are analyzed and output to downstream equipment, and the system's splicing accuracy and single frame time are monitored in real time, with automatic alarms when the limit is exceeded.
[0007] Preferably, the FPGA synchronization pulse controls the exposure timing deviation of each camera to be ≤100ns, and the multiple 3D cameras are 2 to 8 homogeneous or heterogeneous structured light 3D cameras. The data from each path is transmitted to the edge GPU computing unit through the PCIe high-speed parallel channel.
[0008] Preferably, the global joint optimization objective function is: in, For the total number of cameras, The total number of target feature points. For the first The projection function of each camera. This is the trajectory function of the target feature points in the global coordinate system. and The first The rotation matrix and translation vector of each camera. For the first Timing offset of each camera, For the first The first camera observed the first Pixel coordinates of each target feature point is the regularization coefficient; the registration convergence accuracy of global joint calibration is ≤0.03mm, and the trigger threshold for online self-calibration is global stitching error ≥0.05mm.
[0009] Preferably, the geometric feature is a Fast Point Feature Histogram (FPFH), the texture feature is a SURF-accelerated robust feature, and the rigid transformation consistency test adopts the RANSAC random sampling consistency algorithm, which calculates candidate rigid transformations by randomly selecting 3 sets of matching points and selects the optimal transformation with the highest proportion of interior points to exclude mismatched pairs. Based on the CUDA architecture, coarse registration is performed in blocks and in parallel, and the total time for coarse registration of a four-channel camera array is ≤5ms.
[0010] Preferably, the objective function for the weighted point-to-plane distance is: in, The total number of point pairs participating in registration. The first point in the source cloud One registration point to be matched In the target point cloud The nearest point for The unit normal vector at that location, The depth confidence weights are constructed jointly by the shooting distance and the angle of incidence. For Huber robust kernel function: in, The residual threshold parameter is used; the iterative reweighted least squares method uses the average distance of the point cloud ≤ 0.03 mm as the convergence termination condition.
[0011] Preferably, the adaptive voxel block compression uses large-size voxel downsampling for flat, large-area regions, and retains the original high-density point cloud for the edge and corner feature regions of the workpiece. After compression, the global point cloud data volume is 20% to 35% of the total amount of the original multi-source point cloud.
[0012] Preferably, the single-frame time for the lightweight fine-tuning is ≤1.5ms; under the collaborative constraints of parallel coarse registration, fine fusion and cache reuse, the total single-frame time for the entire multi-camera stitched 3D reconstruction is ≤25ms, the global stitching error is ≤0.03mm, and the stitching error fluctuation after 10,000 consecutive reconstructions is ≤0.01mm.
[0013] Secondly, this application provides a global calibration and fusion system for multi-camera 3D stitching, including: The multi-camera synchronous acquisition module is used to trigger multiple 3D cameras to expose synchronously with nanosecond precision through FPGA synchronization pulses, acquire depth point clouds and 2D texture images output by each camera in parallel, and perform distortion correction and invalid depth point removal on the point cloud of a single camera. The global joint calibration module is connected to the multi-camera synchronous acquisition module. It is used to construct a global joint optimization objective function containing the intrinsic parameters, extrinsic parameters and time series offsets of each camera based on the acquired cross-field feature points and solve it synchronously and iteratively. It completes the global coordinate transfer between non-overlapping fields of view through intermediate transition cameras, outputs the global transformation matrix of each camera relative to the world coordinate system, and periodically corrects the extrinsic parameters based on the real-time extracted workpiece geometric features to compensate for calibration drift. The GPU parallel hierarchical coarse registration module is connected to the global joint calibration module. It is used to extract a hybrid descriptor of geometric and texture features from each point cloud simultaneously using the global transformation matrix as the initial mapping. After removing mismatched point pairs through rigid transformation consistency check, the global space is divided into three-dimensional voxel blocks and allocated to different computing cores of the GPU to perform cross-camera feature coarse matching in parallel, thus completing the initial global alignment of multi-view point clouds. The confidence-weighted fine fusion module is connected to the GPU parallel hierarchical coarse registration module. It is used to take the coarsely registered aligned point cloud as input, calculate the depth confidence weight of each 3D point according to the shooting distance and incident angle of each camera, construct a weighted point-to-plane distance objective function and embed a robust kernel function, optimize and eliminate residual misalignment by iterative reweighted least squares method, retain high confidence points in the overlapping area by confidence-weighted fusion, and then use an adaptive voxel block strategy to compress and output the global point cloud. The pre-calibration cache reuse module is connected to the global joint calibration module and the confidence weighted fine fusion module respectively. It is used to store the global calibration parameters and the complete point cloud template of the standard workpiece in a high-speed cache. When processing batch workpieces of the same specification, the cached data is directly called. Only the small deviation between the current frame and the template is lightly fine-tuned, and the cache is automatically updated based on the splicing error monitoring results. The global data output interaction module is connected to the confidence-weighted fine fusion module. It is used to uniformly convert the fused global dense point cloud to the world coordinate system, analyze the three-dimensional pose and size data of the workpiece and output it to downstream equipment, and synchronously monitor the system splicing accuracy and single frame consumption in real time, and automatically alarm when the limit is exceeded.
[0014] Thirdly, this application provides a computer device, including: a memory and a processor, which are interconnected and communicate with each other. The memory stores computer instructions, and the processor executes the computer instructions to realize the global calibration and fusion method for multi-camera 3D stitching of the first aspect.
[0015] Fourthly, this application provides a computer-readable storage medium storing computer instructions, which, when executed by a processor, implement the global calibration and fusion method for multi-camera 3D stitching of the first aspect.
[0016] The global calibration and fusion method, system, and storage medium for multi-camera 3D stitching provided in this application have at least the following beneficial effects: This application effectively solves the core problems of existing multi-camera stitching schemes, such as poor synchronization, difficult calibration, slow stitching, low accuracy, and serious redundant calculations, through systematic innovations in acquisition synchronization, integrated calibration, parallel registration, fusion confidence, and computation caching.
[0017] FPGA hardware nanosecond-level synchronization fundamentally eliminates timing misalignment of moving workpieces. Global joint calibration optimizes timing offsets and extrinsic parameters, achieving micrometer-level accuracy in the calibration process. An online self-calibration mechanism ensures long-term operational stability. GPU parallel coarse registration and point-to-plane weighted fusion significantly improve computational efficiency and convergence speed. Cache reuse minimizes single-frame fine-tuning time in mass production, ensuring the system matches the pace of high-speed production lines. Hybrid feature design and confidence weighting mechanisms make this method robust to weak textures, high reflectivity, and low-confidence noise, resulting in a seamless, hole-free global point cloud after fusion. The entire method can be stably deployed in high-end industrial scenarios such as online inspection of large workpieces and robot grasping guidance, balancing large field-of-view coverage, micrometer-level accuracy, and real-time performance. Attached Figure Description
[0018] Figure 1 A schematic diagram of a global calibration and fusion method for multi-camera 3D stitching is shown; Figure 2 A schematic diagram of a global calibration and fusion system for multi-camera 3D stitching is shown. Detailed Implementation
[0019] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.
[0020] Example 1 See Figure 1 As shown, this application provides a global calibration and fusion method for multi-camera 3D stitching, including the following steps.
[0021] S1. Multiple 3D cameras are triggered to expose synchronously with nanosecond precision by FPGA synchronization pulse, and the depth point cloud and two-dimensional texture image output by each camera are acquired in parallel. Distortion correction and invalid depth point removal are performed on the point cloud of a single camera. S2. Based on the collected cross-field feature points, construct a global joint optimization objective function containing the intrinsic parameters, extrinsic parameters, and time-series offsets of each camera and solve it synchronously and iteratively. Complete the global coordinate transfer between non-overlapping fields of view through intermediate transition cameras, output the global transformation matrix of each camera relative to the world coordinate system, and periodically correct the extrinsic parameters based on the real-time extracted workpiece geometric features to compensate for calibration drift. S3. Using the global transformation matrix as the initial mapping, a hybrid descriptor of geometric and texture features is extracted synchronously for each point cloud. After removing mismatched point pairs through rigid transformation consistency check, the global space is divided into three-dimensional voxel blocks and allocated to different computing cores of the GPU to perform cross-camera feature coarse matching in parallel, thus completing the initial global alignment of multi-view point clouds. S4. Using the coarsely registered aligned point cloud as input, calculate the depth confidence weight of each 3D point based on the shooting distance and incident angle of each camera, construct a weighted point-to-plane distance objective function and embed a robust kernel function, optimize and eliminate residual misalignment through iterative reweighted least squares method, and retain high confidence points by weighted fusion of overlapping areas according to confidence level. Then, use an adaptive voxel block strategy to compress and output the global point cloud. S5. Store the global calibration parameters and the complete point cloud template of the standard workpiece in a high-speed cache. When processing a batch of workpieces of the same specification, the cached data is directly called. Only the minor deviation between the current frame and the template is lightly fine-tuned, and the cache is automatically updated based on the splicing error monitoring results. S6. The fused global dense point cloud is uniformly converted to the world coordinate system, the workpiece's three-dimensional pose and size data are analyzed and output to downstream equipment, and the system splicing accuracy and single frame time are monitored in real time, with automatic alarm when the limit is exceeded.
[0022] In this application, step S1 addresses the spatial misalignment problem caused by insufficient software synchronization accuracy in traditional multi-camera systems from the data acquisition source. Specifically, an array of two or more structured light 3D cameras is constructed, with the cameras distributed around the workpiece to be measured. Each camera receives nanosecond-level synchronization pulse signals through an FPGA hardware synchronization trigger unit. The FPGA directly outputs trigger signals to the external trigger interfaces of each camera through hardware logic gates, unaffected by operation scheduling jitter and network latency, controlling the timing deviation of each camera's exposure start time to the nanosecond level. After synchronized exposure by each camera, each camera outputs the original depth point cloud and the corresponding 2D color texture image. Each data stream is independently transmitted to the edge GPU computing unit through a PCIe high-speed parallel channel. At the GPU, distortion correction is performed on each point cloud, eliminating depth value offsets caused by radial and tangential distortion of the lens based on the distortion coefficients in the camera's intrinsic parameters; simultaneously, invalid depth points are removed, including edge noise with a depth value of zero, exceeding the effective range, and too low confidence. After preprocessing, the local original point cloud sets of each camera are output as input data for subsequent calibration or reconstruction.
[0023] Step S2 establishes a unified spatial reference for the multi-camera array, eliminating ambiguities in the transformation between independent coordinate systems. Specifically, a high-precision 3D spherical target is used as the calibration reference. The 3D spherical target can be observed from any angle by a single camera, and the coordinates of the sphere's center can be extracted. This allows each camera to capture images of the target from different positions, without requiring all cameras to have a common overlapping field of view. This is particularly suitable for calibration scenarios where adjacent cameras in a large-span, wraparound layout have no overlapping areas. During calibration, the target is placed sequentially at multiple positions within the field of view of each camera, and cross-field-of-view feature points are collected simultaneously. A global joint optimization objective function containing all camera intrinsic parameters, extrinsic parameters, and temporal offsets is constructed. The temporal offset is included as an explicit optimization variable in the global cost function and solved iteratively with the extrinsic parameters. In traditional schemes, the temporal offset is only passively compensated through post-registration, while this method decouples the temporal error from the extrinsic parameter estimation from the calibration stage. By solving all parameters simultaneously through a global optimization algorithm, the consistency of extrinsic parameter estimation is significantly improved compared to traditional step-by-step calibration. For cameras with non-overlapping fields of view, coordinate transfer is completed through intermediate transition cameras, and the final output is the global transformation matrix of all cameras relative to a unified world coordinate system. During production line operation, the stable geometric features of the workpiece itself are extracted in real time and compared with external parameters. When the splicing error reaches the set threshold, online self-calibration is automatically triggered to compensate for calibration drift caused by temperature and vibration, without the need for manual calibration during production stoppage.
[0024] Step S3 utilizes the global transformation matrix output from calibration to perform rapid initial alignment of the multi-viewpoint clouds, providing good initial values for fine registration. Specifically, using the global transformation matrix output from step S2 as the initial mapping, a coarse coordinate transformation is performed on each point cloud, causing the point clouds from each viewpoint to initially cluster in the global space. Based on this, a hybrid feature descriptor is simultaneously extracted from each point cloud—geometric features employ FPFH (Fast Point Feature Histogram) to statistically encode the geometric distribution of the local neighborhood of the point cloud; texture features employ SURF (Speed-Up Robust Feature Rendering) to extract local texture distribution information from synchronously acquired 2D texture images. The two features are fused to form a hybrid descriptor, enabling the extraction of effective matching features for various workpieces, both geometrically rich but texture-poor and texture-rich but geometrically flat. After extraction, FLANN (Fast Nearest Neighbor Search) is used for cross-camera feature matching to generate an initial matching set. Subsequently, the RANSAC random sampling consensus algorithm is used to verify the consistency of rigid transformations. Multiple sets of matching point pairs are randomly selected from the matching set to calculate candidate rigid transformations. The proportion of interior points that meet the distance threshold is statistically analyzed to select the optimal transformation and eliminate mismatched point pairs caused by duplicate textures, occluded edges, noise, etc. In the parallel computing architecture, the global space is uniformly divided into three-dimensional voxel blocks. The registration tasks within each voxel block are independent and distributed to different CUDA cores of the GPU for synchronous execution, changing the overall coarse registration from serial to parallel and achieving millisecond-level processing time.
[0025] Step S4 eliminates residual misalignments through high-precision iterative optimization and outputs a high-quality global point cloud through weighted fusion and adaptive compression. Specifically, the depth confidence weight of each 3D point is first calculated, constructed jointly by the shooting distance and the incident angle—the closer the distance and the closer to perpendicular shooting, the higher the depth measurement signal-to-noise ratio and the larger the weight; conversely, the weight is suppressed. In the fine registration stage, a weighted point-to-plane distance objective function is constructed. Unlike traditional point-to-point ICP which directly calculates the Euclidean distance between two points, the point-to-plane metric calculates the normal distance from the source point cloud to the tangent plane of the target point cloud surface after transformation. This normal vector information guides the registration, significantly improving the convergence speed. The objective function embeds depth confidence weights, allowing high-precision points to dominate the optimization direction. A robust kernel function is also introduced—when the registration residual is less than a threshold, squared loss is used for rapid convergence; when it is greater than the threshold, linear loss is switched to suppress outlier interference. The solution is obtained through iterative reweighted least squares, iterating until the average point cloud distance converges to micrometer-level accuracy. During overlapping region fusion, multi-source points falling into the same local neighborhood are weighted according to their confidence level. High-confidence points retain their original coordinates, while low-confidence points are discarded or moved closer to higher-confidence points. At the same time, holes caused by occlusion or reflection are filled. Finally, adaptive voxel block compression is performed: large-size voxel downsampling is used for flat, large-area regions to significantly reduce redundant points, while the original high-density point cloud is retained for workpiece edges and angular feature regions, achieving a balance between data volume reduction and the preservation of key feature accuracy.
[0026] Step S5 avoids repeatedly executing the complete calibration and stitching process in batch production of the same specifications through a cache reuse mechanism. Specifically, after the initial global joint calibration, all camera intrinsic and extrinsic parameters, global transformation matrices, temporal offsets, and other calibration data are stored in a high-speed cache on the GPU. In batch production, coordinate transformation directly reads the cached parameters, eliminating the need for repeated calibration. For workpieces of the same specifications, a standard global stitching point cloud template is generated and cached during the initial processing. Subsequent new frames only need to be matched with the template to calculate minor deviations, and the registration parameters are lightly fine-tuned to output the results; single-frame fine-tuning takes very little time. When the stitching error exceeds a threshold, a cache update is automatically triggered, and calibration or template reconstruction is re-executed.
[0027] Step S6 converts the fused global point cloud into standardized data usable by downstream devices and monitors its status in real time. Specifically, the dense global point cloud is uniformly converted to the world coordinate system, and the workpiece's six-degree-of-freedom pose and key dimension data are analyzed using methods such as principal component analysis or registration to a CAD model. The data is then output to the robot controller, online inspection equipment, or warehouse modeling system via standard industrial communication protocols. A built-in real-time monitoring thread continuously records operating indicators such as global stitching error and single-frame reconstruction time, and automatically alarms when thresholds are exceeded.
[0028] The six interconnected steps address the core technical challenge of existing multi-camera stitching solutions in achieving high precision, high speed, and high robustness in dynamic industrial production lines. FPGA hardware nanosecond-level synchronization eliminates the root cause of timing misalignment in moving workpieces at the acquisition source. Deviations of milliseconds to tens of milliseconds in traditional software synchronization are compressed to the nanosecond level. Spatial misalignment of high-speed moving workpieces in the multi-camera field of view on the production line is essentially negligible, fundamentally resolving the inherent defect that timing deviations cannot be compensated for through subsequent registration. Global joint calibration incorporates the timing offsets of each camera as explicit optimization variables into a unified objective function, iteratively solving the problem synchronously with extrinsic parameters. This completely decouples timing errors from extrinsic parameter estimation, enabling micrometer-level accuracy in the calibration process. The use of a 3D spherical target breaks the layout limitation of traditional checkerboard calibration, which requires all cameras to have a common overlapping field of view, making global calibration of large-span, encircling camera arrays possible. The online self-calibration mechanism utilizes the stable geometric features of the workpiece itself to monitor the calibration status in real time, eliminating the need for manual intervention and ensuring that the system's accuracy does not degrade under harsh conditions such as temperature changes and mechanical creep. GPU parallel hierarchical coarse registration divides the global space into three-dimensional voxel blocks and distributes them to different CUDA cores for synchronous execution, compressing the linear time increase of traditional serial registration to the millisecond level. The fusion of FPFH geometric features and SURF texture features significantly improves the matching success rate for difficult workpieces with weak textures and high reflectivity. Combined with RANSAC rigid transformation consistency checks, it effectively eliminates mismatches, ensuring the reliability of coarse alignment. Confidence-weighted fine fusion uses a point-to-plane distance metric instead of the traditional point-to-point metric, utilizing surface normal vector information to guide registration, significantly improving iterative convergence speed. The embedding of a robust kernel function makes the algorithm insensitive to outliers such as occluded edges and welding arc light. The cache reuse mechanism allows for lightweight fine-tuning of each frame in batch production of the same specifications, requiring only microseconds to milliseconds to output results, greatly reducing redundant computation overhead. Hybrid feature design enables this method to be applicable to various workpieces with rich or scarce geometric features and clear or blurry textures, overcoming the limitation of traditional ICP algorithms in failing to register weakly textured workpieces. The confidence weighting mechanism effectively suppresses the spread of low-confidence noise introduced by long-distance, large-angle oblique photography to high-precision areas. After weighted fusion of overlapping areas, the global point cloud is free of ghosting, holes, and layering faults. Adaptive voxel block compression significantly reduces the amount of data while ensuring the accuracy of edge corners. Through systematic innovations in acquisition synchronization, integrated calibration, parallel registration, confidence-based fusion, and computation caching, the entire method can be stably deployed in high-end industrial scenarios such as online dimensional inspection of large workpieces, large-scale robot grasping and guidance, and precision monitoring of vehicle welding. It effectively solves the systemic problems of poor synchronization, difficult calibration, slow stitching, low fusion accuracy, and serious repetitive calculations in existing multi-camera stitching solutions in actual production line applications.
[0029] Preferably, the exposure timing deviation of each camera is ≤100ns under the control of the FPGA synchronous pulse, and the multiple 3D cameras are 2 to 8 homogeneous or heterogeneous structured light 3D cameras. The data from each path is transmitted to the edge GPU computing unit through the PCIe high-speed parallel channel.
[0030] Specifically, the FPGA synchronization pulse control ensures that the exposure timing deviation of each camera is ≤100ns. This value is far superior to the accuracy level of 5ms~20ms of traditional software synchronization, guaranteeing the spatial position consistency of multiple cameras capturing moving workpieces at the hardware level and eliminating misalignment errors that cannot be compensated for in subsequent registration.
[0031] Multiple 3D cameras can be 2 to 8 homogeneous or heterogeneous structured light 3D cameras. This range covers the most common camera array size in industrial scenarios (4 to 6 cameras are the typical configuration), while also being compatible with the mixed use of homogeneous and heterogeneous cameras, providing flexible hardware selection capabilities to adapt to different field-of-view coverage requirements and cost considerations.
[0032] Data from each stream is transmitted to the edge GPU computing unit via a high-speed parallel PCIe channel. The PCIe channel features high bandwidth and low latency, supporting the parallel, non-blocking transmission of multiple streams of millions of point cloud data to the edge GPU, avoiding data transmission becoming a bottleneck and ensuring the data input rate requirements for subsequent real-time processing.
[0033] By limiting the parameters mentioned above, the timing accuracy, hardware compatibility, and data transmission efficiency of the multi-camera synchronous acquisition process are ensured. Exposure deviation ≤100ns makes the spatial misalignment of moving workpieces negligible (calculated at a speed of 100mm / s, the misalignment is only on the order of 0.01μm), far exceeding the accuracy requirements of precision inspection. A range of 2 to 8 units covers the splicing needs of most large-scale industrial scenarios, and the homogeneous / heterogeneous compatibility design reduces deployment costs and hardware selection constraints. PCIe parallel transmission ensures that multiple massive amounts of data reach the GPU in real time, providing input bandwidth assurance for single-frame reconstruction time ≤25ms.
[0034] Preferably, the global joint optimization objective function is: in, For the total number of cameras, The total number of target feature points. For the first The projection function of each camera. This is the trajectory function of the target feature points in the global coordinate system. and The first The rotation matrix and translation vector of each camera. For the first Timing offset of each camera, For the first The first camera observed the first Pixel coordinates of each target feature point is the regularization coefficient; the registration convergence accuracy of global joint calibration is ≤0.03mm, and the trigger threshold for online self-calibration is global stitching error ≥0.05mm.
[0035] Specifically, the objective function includes the temporal offset of each camera. The temporal offset is incorporated into the global cost function as an explicit optimization variable. In traditional schemes, the temporal offset is only considered as an external error and passively compensated through post-registration. However, this method solves for the temporal offset and extrinsic parameters simultaneously and iteratively, so that the temporal error is no longer independently coupled into the rotation matrix. Translation vector In the estimation, the first term of the function measures the feature points of the three-dimensional target after projection by the function. The pixel error after reprojection onto the image plane of each camera, the second term This is a regularization term used to constrain the physical feasibility of time-series offsets and avoid overfitting.
[0036] The objective function simultaneously solves for all camera parameters using global optimization algorithms such as Levenberg-Marquardt, significantly improving the accuracy and consistency of extrinsic parameter estimation compared to traditional step-by-step calibration schemes. Furthermore, it limits the registration convergence accuracy of global joint calibration to ≤0.03mm, and the trigger threshold for online self-calibration to a global stitching error ≥0.05mm. The former characterizes the achievable accuracy level during the calibration phase, while the latter defines the error boundary for re-performing self-calibration. Together, they constitute a closed-loop calibration maintenance mechanism of "accuracy achievement - error triggering - dynamic correction."
[0037] By explicitly incorporating the temporal offset into the global optimization objective function, the coupling interference between temporal error and extrinsic parameter estimation is fundamentally eliminated, resolving the inherent defect in traditional schemes where temporal deviations cannot be compensated for through subsequent registration. The introduction of a regularization term ensures the numerical stability and physical interpretability of the solution. A registration convergence accuracy of ≤0.03mm provides calibration-level accuracy assurance for the final global stitching quality, while a self-calibration trigger threshold of 0.05mm ensures that the accuracy does not decay during long-term production line operation. Experimental data showing stitching error fluctuations of ≤0.01mm after 10,000 consecutive runs validates the long-term stability of this mechanism.
[0038] Preferably, the geometric feature is the FPFH fast point feature histogram, the texture feature is the SURF accelerated robust feature, and the rigid transformation consistency test adopts the RANSAC random sampling consistency algorithm. The candidate rigid transformation is calculated by randomly selecting 3 sets of matching points and the optimal transformation with the highest proportion of interior points is selected to eliminate false matching pairs. Based on the CUDA architecture, coarse registration is performed in blocks and in parallel, and the total time for coarse registration of a four-channel camera array is ≤5ms.
[0039] Specifically, the geometric features employ FPFH (Fast Point Feature Histogram), which describes the geometric distribution of the neighborhood of each point in the point cloud and is robust to changes in point cloud density and noise. The texture features utilize SURF (Speeded Up Robust Features), extracted from synchronously acquired 2D texture images, providing supplementary matching information for regions with weak geometric features but rich texture. The hybrid descriptor formed by fusing these two features balances geometric structure constraints and texture appearance constraints, significantly improving the matching success rate of workpieces with low geometric features.
[0040] The rigid transformation consistency check uses the RANSAC (Random Sample Consensus) algorithm. Candidate rigid transformations are calculated by randomly selecting 3 sets of matching point pairs, and the proportion of interior points that meet the preset distance threshold is statistically analyzed. The transformation with the highest proportion of interior points is iteratively selected as the optimal output, which effectively eliminates mismatched pairs caused by noise, occlusion, or duplicate textures, and prevents a single erroneous match from destroying the global coarse alignment.
[0041] Based on the CUDA architecture, the global space is divided into 3D voxel blocks and distributed to different GPU computing cores for parallel coarse registration. The matching tasks within each voxel block do not interfere with each other, achieving true parallel acceleration. The total time for coarse registration of the camera array is limited to ≤5ms, and a quantitative indicator of the real-time performance of the coarse registration stage is given.
[0042] By employing a hybrid feature design combining FPFH and SURF, this method becomes applicable to various industrial workpieces with diverse geometries (rich / poor) and textures (clear / fuzzy), overcoming the limitations of traditional ICP which relies solely on geometric distance. This addresses the industry pain point of registration failures for weakly textured workpieces such as metal parts and black plastic parts. RANSAC conformance testing minimizes the risk of coarse registration failures introduced by mismatches, ensuring the reliability of coarse alignment. The CUDA block-parallel strategy breaks through the bottleneck of linearly increasing coarse registration time with the number of cameras in serial schemes (four-camera serial registration typically ≥20ms), compressing the time to less than 5ms, providing core support for real-time targets with ≤25ms.
[0043] Preferably, the objective function for the weighted point-to-plane distance is: in, The total number of point pairs participating in registration. The first point in the source cloud One registration point to be matched In the target point cloud The nearest point for The unit normal vector at that location, The depth confidence weights are constructed jointly by the shooting distance and the angle of incidence. For Huber robust kernel function: in, The residual threshold parameter is used; the iterative reweighted least squares method terminates with the average point cloud distance ≤ 0.03 mm as the convergence condition.
[0044] Specifically, the "point-to-plane" distance replaces the traditional "point-to-point" distance. Traditional ICP uses the Euclidean distance between two points as an error metric, and convergence is slow when there is slippage between the two point clouds along the tangential direction of the plane. This method uses the source point... After transformation to the target point The normal distance of the tangent plane is used as the error term, utilizing the plane normal vector. The provided geometric constraints guide the registration to converge quickly along the normal direction, significantly reducing the number of iterations.
[0045] Confidence weight The introduction of The depth measurement signal-to-noise ratio is constructed jointly by the shooting distance and incident angle of each camera—the closer the distance and the closer the incident angle is to perpendicular shooting, the higher the signal-to-noise ratio of the point, and the higher its weight. This makes high-precision points taken at close range and from the front dominate the registration optimization, while the contribution of low-confidence points taken at long distance and from an angle to the objective function is actively suppressed.
[0046] Huber robust kernel function Embedding, when the registration residual is less than a threshold The quadratic loss is used for fast convergence, and when the value exceeds the threshold, it switches to linear loss to suppress outlier gradient interference, effectively resisting the pull of outlier noise points caused by occlusion, reflection and other factors on the optimization direction.
[0047] The Iterative Reweighted Least Squares (IRLS) method recalculates the weights based on the current residuals in each iteration, optimizing round by round until the average distance of the point cloud converges to ≤0.03mm.
[0048] The weighted point-to-plane distance objective function brings triple gains: the point-to-plane constraint improves the convergence speed by about 40% compared to traditional point-to-point ICP, and significantly reduces the number of iterations; the confidence weight mechanism ensures that high-precision measurement data dominates the registration optimization, guaranteeing global stitching accuracy (≤0.03mm); and the Huber kernel function gives the algorithm strong anti-interference capabilities against occluded edges and reflective noise. The synergy of these three elements enables the fine fusion process to achieve an optimal balance between accuracy, speed, and robustness, laying the algorithmic foundation for the final seamless, ghost-free output of the global point cloud.
[0049] Preferably, adaptive voxel block compression uses large-size voxel downsampling for flat, large-area regions, while retaining the original high-density point cloud for the edges and corners of the workpiece. After compression, the global point cloud data volume is 20% to 35% of the total amount of the original multi-source point cloud.
[0050] Specifically, the adaptive voxel-based block compression strategy and its compression performance metrics are defined. This adaptive compression strategy does not uniformly apply voxel downsampling to the global point cloud at the same scale. Instead, it differentiates based on the density of local geometric features: for flat, large-area regions (such as sheet metal planes and sheet metal surfaces), large-size voxel downsampling is used, retaining only one representative point within each voxel, significantly reducing redundant data; for workpiece edges and angular feature regions, the original high-density point cloud is retained, ensuring that the accuracy of key geometric features is not lost. The compression performance is defined as the compressed global point cloud data volume being 20%–35% of the sum of the original multi-source point clouds. This proportion significantly reduces the burden of subsequent storage, transmission, and pose calculation while ensuring the detail integrity of key feature regions.
[0051] An adaptive block compression strategy achieves an optimal balance between preserving accuracy and reducing data volume. Compared to traditional uniform downsampling (which results in severe loss of accuracy in feature regions) or no compression (which leads to an explosion in data volume), this method dynamically adjusts the compression ratio based on geometric feature density. While ensuring the original accuracy of key detection areas such as edges and corners, it compresses the overall data volume to 1 / 5 to 1 / 3 of the original amount. This not only significantly reduces downstream storage and transmission overhead but also directly accelerates subsequent real-time processing steps such as pose calculation and size measurement. It is an indispensable optimization step for a single-frame reconstruction time of ≤25ms.
[0052] Preferably, the time taken for a single frame of lightweight fine-tuning is ≤1.5ms; under the collaborative constraints of parallel coarse registration, fine fusion and cache reuse, the total time taken for a single frame of the entire multi-camera stitched 3D reconstruction is ≤25ms, the global stitching error is ≤0.03mm, and the stitching error fluctuation after 10,000 consecutive reconstructions is ≤0.01mm.
[0053] Specifically, three levels are defined: First, the time taken for fine-tuning a single frame of the workpiece in the cache reuse scenario is ≤1.5ms, quantifying the lightweight fine-tuning efficiency of the fifth step of cache reuse, indicating that in the batch production mode with cache hits, new frame processing adds almost no additional computational burden. Second, under the synergistic constraints of parallel coarse registration, fine fusion, and cache reuse, the total time taken for a single frame of the entire multi-camera stitched 3D reconstruction is ≤25ms. This indicator is the cumulative result of the aforementioned real-time stages (parallel acquisition and transmission, GPU coarse registration ≤5ms, weighted ICP fast convergence, and cache reuse ≤1.5ms), and is a comprehensive quantification of the computational efficiency of the entire method. Third, the global stitching error is ≤0.03mm, and the stitching error fluctuation after 10,000 consecutive reconstructions is ≤0.01mm. The former characterizes the absolute splicing accuracy of the system, while the latter characterizes the long-term operational stability of the system. 10,000 consecutive reconstructions correspond to a continuous operation cycle of several hours to several days on an industrial production line. A fluctuation range of ≤0.01mm proves that the online self-calibration mechanism effectively compensates for calibration drift introduced by environmental factors such as temperature and vibration.
[0054] By comprehensively defining the three core indicators of accuracy, speed, and stability, the total time per frame is ≤25ms, which can match the cycle time of high-speed production lines of more than 30 pieces per minute (far superior to the traditional serial solution of ≥80ms), breaking through the real-time bottleneck of large-scene 3D reconstruction in industrial online scenarios. The global stitching error is ≤0.03mm, covering the micron-level application requirements of precision assembly and high-precision dimensional inspection. The error fluctuation of ≤0.01mm after 10,000 consecutive reconstructions verifies the actual effectiveness of the online self-calibration mechanism, proving that the accuracy does not decay during long-term continuous operation and has the reliability and stability required for industrial deployment. The simultaneous achievement of the three indicators signifies that this method simultaneously meets the composite requirements of high precision, high speed, and high stability in industrial dynamic production lines, solving the core technical contradiction that existing multi-camera stitching solutions cannot simultaneously achieve these three goals.
[0055] For example, taking a car chassis handling scenario, the workpiece dimensions are approximately 1800mm × 1200mm × 400mm, a typical large-scale, low-texture metal workpiece. Four structured light 3D cameras are arranged around the chassis, and the four cameras are synchronously exposed via an FPGA hardware synchronization trigger unit, with a measured timing deviation of 78ns. The four cameras synchronously output depth point clouds and color texture images, which are transmitted to the RTX 4090 GPU edge computing unit via the PCIe channel. After distortion correction and invalid point removal for each camera point cloud, the images proceed to subsequent processing. The first run performs global joint calibration, using a high-precision 3D spherical target to traverse the entire field of view of the four cameras, constructing a global joint optimization objective function to synchronously solve for intrinsic parameters, extrinsic parameters, and timing offsets. For two sets of cameras arranged opposite each other with non-overlapping fields of view, coordinate transfer is completed through the intermediate adjacent camera. After calibration, the initial mapping deviation is 0.024mm, and the calibration parameters are stored in the cache. After acquiring point clouds for each frame, the global transformation matrix in the cache is used as the initial mapping. Four-way point clouds are simultaneously extracted for FPFH geometric features and SURF texture blending features, with a single-way extraction time of 2.7ms. False matches are removed after RANSAC consistency testing. The global space is divided into 12 3D voxel blocks and distributed to different GPU cores for parallel coarse registration, with a total time of 4.1ms. Subsequently, the depth confidence weights of each point are calculated, a weighted point-to-plane distance objective function is constructed and embedded with the Huber kernel function, and optimized using iterative reweighted least squares. After convergence, the average point cloud distance is 0.018mm. Overlapping regions are fused using confidence weighting. Large-size voxel downsampling is used for flat sheet metal areas, while high-density points are retained at weld edges. After compression, the global point cloud is 27% of the original total, with fine fusion and compression taking 9.3ms. In mass production, cached calibration parameters and a standard chassis template are directly read, with only minor deviation adjustments made for the current frame, taking 1.2ms. The total time for complete reconstruction of a single frame is 21.3ms. The global point cloud is converted to the robot coordinate system, and a six-DOF pose is output to guide the robot in grasping. After 8 hours of continuous operation, the stitching error fluctuation is ≤0.008mm, meeting the production line's cycle time requirement of 25 pieces / minute.
[0056] For example, taking the online dimensional inspection scenario of a large sheet metal box as an example, the box size is 1200mm×800mm×600mm, and the production line inspection accuracy requirement is ±0.05mm. Six high-resolution structured light 3D cameras are widely distributed around the box and on top, with less than 15% overlap between adjacent cameras and no overlap between some opposing cameras. The FPGA synchronously triggers the simultaneous exposure of the six cameras with a timing deviation of 86ns, and the six point cloud and texture images are transmitted to the GPU. During the calibration stage, due to the extremely small overlap between adjacent cameras, the traditional checkerboard method is not applicable. Three-dimensional spherical targets are placed in the field of view of each camera and the small overlap transition area, and the coordinate transfer between non-overlapping opposing cameras is completed by connecting intermediate transition cameras. All parameters are solved synchronously through global joint optimization, and the initial mapping error after calibration is 0.026mm. In the coarse registration stage, the hybrid feature dimension was increased to 384, enhancing the feature representation of sheet metal edges and corner areas. After simultaneous extraction of point clouds from all sources, FLANN matching and RANSAC verification were performed within overlapping small areas. Coarse registration was performed in parallel using 18 voxel blocks, taking 4.8ms. In the fine fusion stage, to address depth noise caused by localized reflections on the painted sheet metal, the confidence weight of areas with abnormal depth gradients was automatically reduced to suppress noise propagation. The weighted ICP converged to 0.018mm after 8 iterations. The data volume after adaptive compression was 21.7% of the original total, with fusion compression taking 10.5ms. Fine-tuning after reusing the cached template took 1.4ms, and the total time for complete reconstruction of a single frame was 23.7ms. After outputting the global point cloud to the world coordinate system, the length, width, height, diagonal, and edge gap measurements were automatically completed. Verification by three-coordinate comparison showed a measurement deviation ≤ ±0.03mm, meeting the ±0.05mm detection tolerance. This embodiment verifies that even in an extreme layout where the overlap field of view between adjacent cameras is extremely small (<15%), the method can still achieve micron-level online full-size detection of large-scale sheet metal boxes.
[0057] Example 2 This embodiment provides a global calibration and fusion system for multi-camera 3D stitching, which is applied to the global calibration and fusion method for multi-camera 3D stitching provided in Embodiment 1 above for illustration. See also Figure 2 As shown, the global calibration and fusion system for multi-camera 3D stitching provided in this application may include the following modules.
[0058] The multi-camera synchronous acquisition module is used to trigger multiple 3D cameras to expose synchronously with nanosecond precision through FPGA synchronization pulses, acquire depth point clouds and 2D texture images output by each camera in parallel, and perform distortion correction and invalid depth point removal on the point cloud of a single camera. The global joint calibration module is connected to the multi-camera synchronous acquisition module. It is used to construct a global joint optimization objective function containing the intrinsic parameters, extrinsic parameters and time series offsets of each camera based on the acquired cross-field feature points and solve it synchronously and iteratively. It completes the global coordinate transfer between non-overlapping fields of view through intermediate transition cameras, outputs the global transformation matrix of each camera relative to the world coordinate system, and periodically corrects the extrinsic parameters based on the real-time extracted workpiece geometric features to compensate for calibration drift. The GPU parallel hierarchical coarse registration module is connected to the global joint calibration module. It is used to extract a hybrid descriptor of geometric and texture features from each point cloud simultaneously using the global transformation matrix as the initial mapping. After removing mismatched point pairs through rigid transformation consistency check, the global space is divided into three-dimensional voxel blocks and distributed to different computing cores of the GPU to perform cross-camera feature coarse matching in parallel, thus completing the initial global alignment of multi-view point clouds. The confidence-weighted fine fusion module is connected to the GPU parallel hierarchical coarse registration module. It is used to take the coarsely registered aligned point cloud as input, calculate the depth confidence weight of each 3D point according to the shooting distance and incident angle of each camera, construct the weighted point-to-plane distance objective function and embed a robust kernel function, optimize and eliminate residual misalignment by iterative reweighted least squares method, and retain high confidence points by confidence-weighted fusion of overlapping areas. Finally, an adaptive voxel block strategy is used to compress and output the global point cloud. The pre-calibration cache reuse module is connected to the global joint calibration module and the confidence weighted fine fusion module respectively. It is used to store the global calibration parameters and the complete point cloud template of the standard workpiece in the high-speed cache. When processing batch workpieces of the same specification, the cached data is directly called. Only the small deviation between the current frame and the template is lightly fine-tuned, and the cache is automatically updated based on the splicing error monitoring results. The global data output interaction module, connected to the confidence-weighted fine fusion module, is used to uniformly convert the fused global dense point cloud to the world coordinate system, analyze the workpiece's three-dimensional pose and size data and output them to downstream equipment, and synchronously monitor the system's stitching accuracy and single frame consumption in real time, and automatically alarm when the limit is exceeded.
[0059] The global calibration and fusion system for multi-camera 3D stitching provided in this application embodiment can be applied to the global calibration and fusion method for multi-camera 3D stitching provided in the above embodiment 1. For relevant details, please refer to the above method embodiment. The implementation principle and technical effect are similar, and will not be repeated here.
[0060] It should be noted that the global calibration and fusion system for multi-camera 3D stitching provided in this embodiment is only illustrated by the above-mentioned division of functional modules / units. In practical applications, the above functions can be assigned to different functional modules / units as needed, that is, the internal structure of the global calibration and fusion system for multi-camera 3D stitching can be divided into different functional modules / units to complete all or part of the functions described above. Furthermore, the implementation method of the global calibration and fusion method for multi-camera 3D stitching provided in the first embodiment above and the implementation method of the global calibration and fusion system for multi-camera 3D stitching provided in this second embodiment belong to the same concept. The specific implementation process of the global calibration and fusion system for multi-camera 3D stitching provided in this second embodiment is detailed in the first embodiment above, and will not be repeated here.
[0061] Example 3 One embodiment of this application also provides a computer device, including but not limited to a processor and a memory, wherein the memory stores computer instructions, and the processor executes the computer instructions to implement the method described above.
[0062] One embodiment of this application also provides a computer-readable storage medium for storing a computer program that, when executed by a processor, implements the method described in the above-described method embodiments.
Claims
1. A global calibration and fusion method for multi-camera 3D stitching, characterized in that, include: Multiple 3D cameras are triggered to expose synchronously with nanosecond precision using FPGA synchronization pulses. The depth point cloud and 2D texture images output by each camera are acquired in parallel, and distortion correction and invalid depth point removal are performed on the point cloud of each camera. Based on the collected cross-field feature points, a global joint optimization objective function containing the intrinsic parameters, extrinsic parameters, and temporal offsets of each camera is constructed and solved iteratively in a synchronous manner. Global coordinate transfer between non-overlapping fields of view is completed through intermediate transition cameras. The global transformation matrix of each camera relative to the world coordinate system is output. The extrinsic parameters are periodically corrected based on the real-time extracted workpiece geometric features to compensate for calibration drift. Using the global transformation matrix as the initial mapping, a hybrid descriptor of geometric and texture features is extracted synchronously for each point cloud. After removing mismatched point pairs through rigid transformation consistency check, the global space is divided into three-dimensional voxel blocks and allocated to different computing cores of the GPU to perform cross-camera feature coarse matching in parallel, thus completing the initial global alignment of multi-view point clouds. Using the coarsely registered aligned point cloud as input, the depth confidence weight of each 3D point is calculated based on the shooting distance and incident angle of each camera. A weighted point-to-plane distance objective function is constructed and a robust kernel function is embedded. The residual misalignment is eliminated by iterative reweighted least squares optimization. Overlapping areas are fused according to confidence level to retain high confidence points. Finally, an adaptive voxel block strategy is used to compress and output the global point cloud. The global calibration parameters and the complete point cloud template of the standard workpiece are stored in a high-speed cache. When processing a batch of workpieces of the same specification, the cached data is directly called. Only the minor deviations between the current frame and the template are lightly fine-tuned, and the cache is automatically updated based on the splicing error monitoring results. The fused global dense point cloud is uniformly converted to the world coordinate system, the workpiece's three-dimensional pose and size data are analyzed and output to downstream equipment, and the system's splicing accuracy and single frame time are monitored in real time, with automatic alarms when the limit is exceeded.
2. The global calibration and fusion method for multi-camera 3D stitching according to claim 1, characterized in that, The FPGA synchronization pulse controls the exposure timing deviation of each camera to ≤100ns. The multiple 3D cameras are 2 to 8 homogeneous or heterogeneous structured light 3D cameras. Data from each path is transmitted to the edge GPU computing unit through the PCIe high-speed parallel channel.
3. The global calibration and fusion method for multi-camera 3D stitching according to claim 2, characterized in that, The global joint optimization objective function is: in, For the total number of cameras, The total number of target feature points. For the first The projection function of each camera. This is the trajectory function of the target feature points in the global coordinate system. and The first The rotation matrix and translation vector of each camera. For the first Timing offset of each camera, For the first The first camera observed the first Pixel coordinates of each target feature point is the regularization coefficient; the registration convergence accuracy of global joint calibration is ≤0.03mm, and the trigger threshold for online self-calibration is global stitching error ≥0.05mm.
4. The global calibration and fusion method for multi-camera 3D stitching according to claim 3, characterized in that: The geometric features are FPFH fast point feature histograms, and the texture features are SURF accelerated robust features. The rigid transformation consistency test uses the RANSAC random sampling consistency algorithm, which calculates candidate rigid transformations by randomly selecting 3 sets of matching points and selects the optimal transformation with the highest proportion of internal points to eliminate false matching pairs. Based on the CUDA architecture, coarse registration is performed in blocks and in parallel, and the total time for coarse registration of a four-channel camera array is ≤5ms.
5. The global calibration and fusion method for multi-camera 3D stitching according to claim 4, characterized in that, The objective function for the weighted point-to-plane distance is: in, The total number of point pairs participating in registration. The first point in the source cloud One registration point to be matched In the target point cloud The nearest point for The unit normal vector at that location, The depth confidence weights are constructed jointly by the shooting distance and the angle of incidence. For Huber robust kernel function: in, The residual threshold parameter is used; the iterative reweighted least squares method uses the average distance of the point cloud ≤ 0.03 mm as the convergence termination condition.
6. The global calibration and fusion method for multi-camera 3D stitching according to claim 5, characterized in that, The adaptive voxel block compression uses large-size voxel downsampling for flat, large-area regions, while retaining the original high-density point cloud for workpiece edges and angular feature regions. After compression, the global point cloud data volume is 20% to 35% of the total amount of the original multi-source point cloud.
7. The global calibration and fusion method for multi-camera 3D stitching according to claim 6, characterized in that, The single-frame time for the lightweight fine-tuning is ≤1.5ms; under the collaborative constraints of parallel coarse registration, fine fusion and cache reuse, the total single-frame time for the entire multi-camera stitched 3D reconstruction is ≤25ms, the global stitching error is ≤0.03mm, and the stitching error fluctuation after 10,000 consecutive reconstructions is ≤0.01mm.
8. A global calibration and fusion system for multi-camera 3D stitching, characterized in that, include: The multi-camera synchronous acquisition module is used to trigger multiple 3D cameras to expose synchronously with nanosecond precision through FPGA synchronization pulses, acquire depth point clouds and 2D texture images output by each camera in parallel, and perform distortion correction and invalid depth point removal on the point cloud of a single camera. The global joint calibration module is connected to the multi-camera synchronous acquisition module. It is used to construct a global joint optimization objective function containing the intrinsic parameters, extrinsic parameters and time series offsets of each camera based on the acquired cross-field feature points and solve it synchronously and iteratively. It completes the global coordinate transfer between non-overlapping fields of view through intermediate transition cameras, outputs the global transformation matrix of each camera relative to the world coordinate system, and periodically corrects the extrinsic parameters based on the real-time extracted workpiece geometric features to compensate for calibration drift. The GPU parallel hierarchical coarse registration module is connected to the global joint calibration module. It is used to extract a hybrid descriptor of geometric and texture features from each point cloud simultaneously using the global transformation matrix as the initial mapping. After removing mismatched point pairs through rigid transformation consistency check, the global space is divided into three-dimensional voxel blocks and allocated to different computing cores of the GPU to perform cross-camera feature coarse matching in parallel, thus completing the initial global alignment of multi-view point clouds. The confidence-weighted fine fusion module is connected to the GPU parallel hierarchical coarse registration module. It is used to take the coarsely registered aligned point cloud as input, calculate the depth confidence weight of each 3D point according to the shooting distance and incident angle of each camera, construct a weighted point-to-plane distance objective function and embed a robust kernel function, optimize and eliminate residual misalignment by iterative reweighted least squares method, retain high confidence points in the overlapping area by confidence-weighted fusion, and then use an adaptive voxel block strategy to compress and output the global point cloud. The pre-calibration cache reuse module is connected to the global joint calibration module and the confidence weighted fine fusion module respectively. It is used to store the global calibration parameters and the complete point cloud template of the standard workpiece in a high-speed cache. When processing batch workpieces of the same specification, the cached data is directly called. Only the small deviation between the current frame and the template is lightly fine-tuned, and the cache is automatically updated based on the splicing error monitoring results. The global data output interaction module is connected to the confidence-weighted fine fusion module. It is used to uniformly convert the fused global dense point cloud to the world coordinate system, analyze the three-dimensional pose and size data of the workpiece and output it to downstream equipment, and synchronously monitor the system splicing accuracy and single frame consumption in real time, and automatically alarm when the limit is exceeded.
9. A computer device, characterized in that, include: The system includes a memory and a processor, which are interconnected. The memory stores computer instructions, and the processor executes these computer instructions to implement the global calibration and fusion method for multi-camera 3D stitching as described in any one of claims 1-7.
10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the global calibration and fusion method for multi-camera 3D stitching as described in any one of claims 1-7.
