Single-lens multi-sensor joint calibration and splicing method based on co-pose constraint

Abstract

The application relates to the technical field of image processing, and particularly discloses a single-lens multi-sensor joint calibration and splicing method based on a co-pose constraint, which comprises the following steps: extracting corner point features in each sub-image, and identifying the identity index of the corner point features on a composite calibration board; establishing a unified imaging geometric model, performing two-stage joint calibration based on the unified imaging geometric model, and the optimization target of the two-stage joint calibration comprises: single field of view reprojection error minimization of each sub-image, and projection consistency constraint between adjacent sub-sensors; determining the relative positions of the sub-images in a global splicing canvas according to the principal point parameters in the solved intrinsic parameters, and splicing a panoramic image. The application establishes a unified imaging model with a co-pose constraint, simultaneously introduces single field of view reprojection and overlapping field of view projection consistency constraints in calibration, makes the multi-sensor system form a unified geometric reference, and significantly reduces the misplacement and ghosting of a splicing boundary.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, specifically to a single-lens multi-sensor joint calibration and stitching method based on co-pose constraints. Background Technology

[0002] Single-lens multi-sensor stitching imaging systems typically utilize the same main lens or common lens optical path to distribute large field-of-view information across multiple sensors for parallel acquisition, thus balancing the requirements of large field-of-view coverage and high-resolution imaging. A key characteristic of this type of system is that multiple sub-fields of view originate from the same imaging link, but at the image output end, they appear as multiple separate sub-images. Therefore, its calibration and stitching differ from both single-camera calibration and the back-end registration of completely independent multi-camera systems.

[0003] Existing calibration and stitching methods for multi-sensor imaging systems include traditional calibration models based on whole-panel observation or complete feature distribution, models that treat each sub-sensor as an independent camera and calibrate it separately, and stitching models that rely on texture features of overlapping regions for back-end registration.

[0004] Traditional calibration models based on whole-plate observation or complete feature distribution typically require a single sensor to completely observe the calibration plate and establish stable constraints based on continuous and sufficient corner point distribution. However, in single-lens multi-sensor systems, a single sub-sensor often can only observe a local area of ​​the calibration plate, making traditional whole-plate observation models difficult to apply directly and prone to problems such as insufficient feature constraints, easy confusion of corner point identities, and decreased calibration stability. Models that treat each sub-sensor as an independent camera and calibrate it separately usually first solve the intrinsic and distortion parameters of each sensor independently, and then restore their relative relationships through subsequent steps. This type of method does not fully utilize the structural characteristics of single-lens multi-sensor systems that share the same physical optical center and the same external pose. Therefore, although the individual reprojection error of each sub-field of view may be small, problems such as inconsistent principal points, inconsistent scales, boundary misalignment, and local ghosting may still occur when stitching the whole system together.

[0005] Stitching models that rely on texture features of overlapping regions for back-end registration typically achieve stitching by extracting image feature points from the overlapping regions and estimating homography, affine relations, or translation amounts. However, for the system targeted by this invention, the overlapping regions of adjacent fields of view are usually small and prone to insufficient texture, duplicate textures, or local illumination differences, resulting in insufficient matching points, increased mismatches, and unstable geometric relationship estimation, making it difficult to meet the requirements for high-precision continuous stitching. Summary of the Invention

[0006] The purpose of this invention is to provide a single-lens multi-sensor joint calibration and stitching method based on co-pose constraints, so as to solve the problems mentioned in the background art.

[0007] To achieve the above objectives, the present invention provides the following technical solution:

[0008] A method for joint calibration and stitching of multiple sensors in a single lens based on co-pose constraints, the method comprising:

[0009] Acquire images from multiple sub-sensors, each containing a local area of ​​the composite calibration plate;

[0010] Extract corner features from each sub-image and identify the identity index of the corner features on the composite calibration board;

[0011] A unified imaging geometry model is established, which describes the multiple sub-sensors as coupled imaging units that share the same external pose and differ only in the arrangement of the focal plane.

[0012] Based on the unified imaging geometry model, a two-stage joint calibration is performed to solve the structural parameters of each sub-sensor that are used to characterize the differences in their focal plane arrangement. The optimization objectives of the two-stage joint calibration include minimizing the single-view reprojection error of each sub-image and the projection consistency constraint of the overlapping fields of view between adjacent sub-sensors.

[0013] Based on the principal point parameters obtained from the solution, the relative positions of each sub-image in the global stitching canvas are determined, and the sub-images are stitched together into a panoramic image accordingly.

[0014] As a further embodiment of the present invention, the composite calibration plate is a ChArUco calibration plate.

[0015] As a further aspect of the present invention, the process of extracting corner features from each sub-image and identifying their identity index includes:

[0016] Detect ArUco markers in sub-images and decode them to obtain their identity index;

[0017] Based on the geometric prior of the ArUco markers, interpolation predicts the initial positions of the checkerboard corner points in the sub-image;

[0018] Sub-pixel refinement is performed in the neighborhood of the initial position to obtain the final coordinates of the chessboard corner points.

[0019] As a further embodiment of the present invention, the two-stage joint calibration specifically includes:

[0020] Phase 1: The initial values ​​of the intrinsic parameters of each sub-sensor are solved by minimizing the single-view reprojection error of the corner features in each sub-image as the optimization objective.

[0021] The second stage: Based on the initial values, the projection consistency constraint of the overlapping field of view between adjacent sub-sensors is introduced, and together with the minimization of the single field of view reprojection error, it is used as the optimization objective to jointly fine-tune the intrinsic parameters and structural parameters.

[0022] As a further aspect of the present invention, the projection consistency constraint of the overlapping field of view is constructed in the following manner:

[0023] Based on the identity index of the composite calibration plate, determine the pairs of feature points with the same name in the overlapping area of ​​adjacent sub-sensors;

[0024] The corresponding feature point pairs are back-projected onto the same reference physical plane where the composite calibration plate is located through the intrinsic parameters of their respective sub-sensors;

[0025] Constraints are constructed based on the geometric positional differences of the two points on the reference physical plane after back projection.

[0026] As a further embodiment of the present invention, the unified imaging geometric model is as follows:

[0027] The multiple sub-sensors share the same rotation matrix and translation vector from the world coordinate system to the camera coordinate system;

[0028] The intrinsic parameter matrix of each sub-sensor contains the coordinates of the principal points used to characterize their positional deviations on the unified physical focal plane.

[0029] As a further aspect of the present invention, the relative position of each sub-image in the global stitching canvas is determined based on the principal point parameter, specifically including:

[0030] Construct a virtual physical plane with the system's optical axis as the origin;

[0031] For each sub-image, calculate the offset vector of its image center relative to its principal point;

[0032] Align the principal points of all sub-images to the origin of the virtual physical plane, and determine their relative positions in the virtual physical plane based on the offset vector of each sub-image;

[0033] The boundary range of the global mosaic canvas is determined based on the relative positions of all sub-images in the virtual physical plane.

[0034] Compared with the prior art, the beneficial effects of the present invention are:

[0035] By establishing a unified imaging model with co-pose constraints and simultaneously introducing consistency constraints between single-field reprojection and overlapping-field projection during calibration, a unified geometric benchmark is formed in the multi-sensor system during the calibration stage, significantly reducing misalignment and ghosting at the stitching boundaries.

[0036] A two-stage joint optimization strategy, from coarse to fine, is adopted. First, the stable initial value of a single field of view is solved, and then multi-field of view constraints are introduced for fine-tuning. This avoids the drift of intrinsic parameters caused by parameter coupling and ensures the physical interpretability of the calibration results.

[0037] Using a composite calibration plate as the preferred feature extraction method under local observation conditions can provide corner point inputs with identity discrimination capabilities for joint calibration, thereby improving the stability of feature extraction and correspondence establishment under local field of view conditions. Attached Figure Description

[0038] To more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention.

[0039] Figure 1 This is a flowchart illustrating a single-lens multi-sensor joint calibration and stitching method based on co-pose constraints provided in an embodiment of the present invention.

[0040] Figure 2 This is a schematic diagram of the imaging geometric model provided in an embodiment of the present invention.

[0041] Figure 3 This is a sub-image to be spliced, provided in an embodiment of the present invention.

[0042] Figure 4 The image provided in this embodiment of the invention shows a stitching effect optimized using only reprojection error.

[0043] Figure 5 This is a stitched diagram showing the effect of multi-objective constraint joint optimization provided in an embodiment of the present invention. Detailed Implementation

[0044] To make the technical problems to be solved, the technical solutions, and the beneficial effects of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the present invention and are not intended to limit the present invention.

[0045] Figure 1 The flowchart illustrates a single-lens multi-sensor joint calibration and stitching method based on co-pose constraints. In this embodiment of the invention, the single-lens multi-sensor joint calibration and stitching method based on co-pose constraints includes:

[0046] Acquire images from multiple sub-sensors, each containing a local area of ​​the composite calibration plate;

[0047] Extract corner features from each sub-image and identify the identity index of the corner features on the composite calibration board;

[0048] A unified imaging geometry model is established, which describes the multiple sub-sensors as coupled imaging units that share the same external pose and differ only in the arrangement of the focal plane.

[0049] Based on the unified imaging geometry model, a two-stage joint calibration is performed to solve the structural parameters of each sub-sensor that are used to characterize the differences in their focal plane arrangement. The optimization objectives of the two-stage joint calibration include minimizing the single-view reprojection error of each sub-image and the projection consistency constraint of the overlapping fields of view between adjacent sub-sensors.

[0050] Based on the principal point parameters obtained from the solution, the relative positions of each sub-image in the global stitching canvas are determined, and the sub-images are stitched together into a panoramic image accordingly.

[0051] In this embodiment, the overall process of the present invention can be summarized into four steps: First, stable corner points in the local field of view are extracted and identified using a composite calibration board; second, a unified imaging geometric model is established by combining the common lens structure of a single-lens multi-sensor system; third, multi-field joint calibration is performed based on this model, and the structural parameters of the participating systems within each sensor are solved simultaneously through phased optimization; finally, an analytical geometric mapping under a unified canvas is directly established based on the calibration results, realizing multi-field image alignment and stitching without relying on subsequent texture registration, such as... Figure 1 As shown.

[0052] This invention uses a composite calibration plate as a spatial reference and combines a joint calibration strategy based on co-pose constraints with a geometric correction method based on imaging mechanism to establish an analytical mapping relationship from the image coordinates of each sensor to a unified global coordinate.

[0053] In a preferred embodiment of the present invention, the composite calibration board is a ChArUco calibration board. The process of extracting corner features from each sub-image and identifying their identity index includes:

[0054] Detect ArUco markers in sub-images and decode them to obtain their identity index;

[0055] Based on the geometric prior of the ArUco markers, interpolation predicts the initial positions of the checkerboard corner points in the sub-image;

[0056] Sub-pixel refinement is performed in the neighborhood of the initial position to obtain the final coordinates of the chessboard corner points.

[0057] In this embodiment, the present invention employs the ChArUco composite calibration plate for calibration plate selection. Its advantage lies not in simply increasing the number of corner points, but in combining the positioning accuracy of the checkerboard corner points with the uniqueness of the ArUco markers, thus making it more suitable for stable detection and matching under local field-of-view conditions. For sensors that can only observe a local area of ​​the calibration plate, this calibration plate can provide reliable index information for cross-field-of-view geometric correlation in subsequent joint calibration while ensuring corner point accuracy.

[0058] In the specific detection process, the input image is first converted to grayscale and binarized, and then candidate ArUco regions are identified through contour extraction and quadrilateral filtering. Subsequently, perspective transformation is used to normalize the candidate regions to a standard square grid, and dictionary matching is used to complete encoding, decoding, and label confirmation. The valid labels obtained after identification provide geometric priors for subsequent checkerboard corner interpolation and sub-pixel refinement.

[0059] Let the input image be I. First, convert it to a grayscale image. Then, use the local average adaptive thresholding method to binarize the image. Its mathematical expression is:

[0060] ;

[0061] In the formula, I(x,y) represents the gray value of the input image at pixel (x,y), where x and y represent the image coordinates; B(x,y) represents the binarization result. represents the local average gray level within window w centered at pixel (x,y); C represents the threshold bias constant; w represents the local statistical window.

[0062] After binarization, an image dilation operation is performed to separate closely adjacent marked regions, ensuring that each black or white region becomes an independent connected component.

[0063] Subsequently, the algorithm extracts all contours from the dilated image and uses a polygon approximation algorithm to select convex quadrilaterals with 4 vertices and an area that meets a preset threshold as candidate regions for labeling. Polygon approximation is then performed on each convex hull to select quadrilateral candidate regions with exactly four vertices.

[0064] Next, construct from the standard unit square Perspective transformation matrix of the quadrilateral By using perspective transformation, the detected arbitrary-shaped quadrilaterals are remapped into standard square images. To eliminate the distortion caused by perspective projection, so that:

[0065] , ;

[0066] In the formula, This represents the perspective transformation matrix corresponding to the k-th candidate quadrilateral region; , [u, v, w] represents the normalized coordinates in a standard unit square. T This represents the mapping result in homogeneous coordinates; This represents the coordinates of the corresponding image point. u and v are the first two dimensions of the homogeneous coordinates, and w is the homogeneous scale component used for perspective division.

[0067] use The marked regions are reverse-mapped to a regular grid, and grayscale values ​​are sampled at the center of each sub-block, resulting in a corrected standard square image. The grid is divided into n×n cells. For each cell, its average gray value is binarized to extract the marker and form a binary bit string. .

[0068] b k With predefined dictionaries Perform Hamming distance matching on all templates in the dataset, where the Hamming distance is d. H The calculation is as follows:

[0069] ;

[0070] In the formula, d H (b,d i ) represents the binary string b and the dictionary template d i Hamming distance between them; b j d represents the binary value of the j-th bit of the encoded string to be recognized. i,j This represents the binary value of the i-th dictionary template at the j-th bit, and n represents the total number of bits in the encoded string.

[0071] Based on the successful identification of ArUco markers, the second stage utilizes the geometric priors of the markers to perform interpolation and sub-pixel refinement of the checkerboard corner points. Let the ChArUco board consist of n... x ×n y The grid consists of k squares, each with a side length of s. The world coordinate system origin is placed at the top-left corner. Then, the world coordinates of the k corner points of the chessboard are... It can be directly calculated from its row and column indices.

[0072] Using the M valid ArUco markers detected in the previous stage, construct a set of marked corner points. Based on these point pairs, the local homography matrix H between the image plane and the calibration plate plane is estimated by minimizing the reprojection error.

[0073] ;

[0074] In the formula, Represents the local homography matrix that minimizes the reprojection error; u i This represents the coordinates of the i-th observed marker point in the image plane; This represents the corresponding calibration plate world coordinates; π(·) represents the projection operator.

[0075] use It can predict the initial position of any corner point of the chessboard to be detected in the image. : ;

[0076] In the formula, This indicates the initial predicted position of the corner point of the chessboard grid to be detected in the image; This indicates the world coordinates of the corner point in the calibration plate coordinate system; Let represent the aforementioned optimal homography matrix.

[0077] This is for perspective division. To obtain sub-pixel precision, at the initial position... The optimization is performed iteratively within the local neighborhood Ω. The goal is to find the precise position q such that this point is orthogonal to the gradient directions of all pixels in its neighborhood, i.e., to minimize the following energy function:

[0078] ;

[0079] In the formula, E(q) represents the energy function for corner refinement; q represents the sub-pixel corner coordinates to be determined; Ω represents the local optimization neighborhood; p represents the pixel in the neighborhood; ∇I(p) represents the image gradient at point p; and w(p) represents the Gaussian weight function.

[0080] In a preferred embodiment of the present invention, the unified imaging geometry model is as follows:

[0081] The multiple sub-sensors share the same rotation matrix and translation vector from the world coordinate system to the camera coordinate system;

[0082] The intrinsic parameter matrix of each sub-sensor contains the principal point coordinates used to characterize the deviation of its arrangement position on a unified physical focal plane. The image points corresponding to the multiple sub-sensors are all imaged by the same main lens and share the same optical center.

[0083] In this embodiment, addressing the problem of "single-channel calibration is possible, but overall stitching is difficult" in multi-sensor systems, this invention does not treat each sensor as an independent free imaging unit, nor does it completely delegate the geometric relationships between multiple fields of view to the back-end image registration process. Instead, it introduces the physical prior of a common-mirror structure, treating the multi-sensor system as a coupled imaging unit that shares a unified external pose and differs only in its arrangement at the focal plane position. This reduces redundant degrees of freedom in parameter solving and provides a unified physical basis for subsequent consistency constraints and analytical geometric stitching of overlapping fields of view.

[0084] Regarding the imaging geometry model, this invention establishes the mapping relationship between the world coordinate system, camera coordinate system, image physical coordinate system, and pixel coordinate system. Based on the structural characteristics of a single-lens confocal array, it uniformly describes each sub-field-of-view sensor within the same imaging framework. Its core significance lies in the fact that although different sensors output different sub-images, they share the same principal optical center and external pose; their geometric differences are mainly reflected in the positional offset on the focal plane. Therefore, the subsequent stitching problem can be transformed into a joint solution of the geometric relationships of each sub-field of view under a unified imaging model, rather than independent post-registration.

[0085] To accurately describe the geometric mapping relationship between 3D spatial points and 2D image pixels in a large field-of-view high-resolution array camera, this section first establishes the imaging geometric model of the system. This system adopts a single-lens multi-sensor architecture, using a prism optical path to project the same optical field of view onto ten sensors. Its physical structure is equivalent to multiple discrete sensor chips arranged discretely on the same physical focal plane, collectively forming a large field-of-view imaging plane. Therefore, each sensor shares the same optical center and external pose in the imaging link; their geometric differences are only reflected in the spatial position offset of each sub-sensor on the focal plane. To quantitatively describe the imaging link, world coordinate systems are defined. Camera coordinate system Pixel coordinate system and image coordinate system .

[0086] Without considering distortion, the imaging geometry model can be represented as a linear model. The imaging geometry model is as follows: Figure 2 As shown, due to its linear relationship, the optical center O C The midpoint P in space and the image point p of P on the image plane are in a straight line relationship.

[0087] The camera coordinate system has its unique origin O at the optical center of the principal lens. C The optical axis direction is Z. CAxis. Since all sensors are fixed to the same rigid body via a prism confocal structure, and the normal vector of the photosensitive surface is parallel to the optical axis, in the ideal model, all field-of-view sensors share the same set of extrinsic parameters. Assume that the homogeneous coordinates of any point P in space in the world coordinate system are... The coordinates of this point are transformed to the camera coordinate system. The rigid body transformation relationship is satisfied as follows:

[0088] ;

[0089] In the formula, P c R represents the coordinates of a point in the camera coordinate system; R represents the rotation matrix from the world coordinate system to the camera coordinate system; t represents the translation vector; [X w ,Y w Z w ] T This represents the coordinates of a point in space within the world coordinate system.

[0090] Where R and t are the rotation matrix and translation vector of the entire array camera relative to the world coordinate system. This shared extrinsic parameter constraint reduces the number of parameters to be optimized during the calibration process and ensures the uniformity of the spatial reference of multiple sensors.

[0091] Subsequently, the process of projecting spatial points from the camera coordinate system onto the two-dimensional image plane is described. This process includes two steps: perspective projection and discretization sampling. An image physical coordinate system (x, y) is established with the intersection of the optical axis and the focal plane (i.e., the ideal principal point o) as the origin. Let the physical focal length of the lens be f. According to the principle of similar triangles, its ideal projection relationship on the physical focal plane is as follows: Figure 2 As shown, the following conditions are met:

[0092] ;

[0093] In the formula, x represents the horizontal coordinate of the point in the image's physical coordinate system; f represents the equivalent focal length of the main lens; X c Y c These represent the horizontal and depth coordinates of a point in the camera coordinate system, respectively.

[0094] ;

[0095] In the formula, y represents the vertical coordinate of the point in the physical coordinate system of the image; Y c Z c These represent the longitudinal and depth coordinates of a point in the camera coordinate system, respectively.

[0096] To transform continuous physical coordinates into discrete digital image coordinates, a pixel coordinate system (u, v) needs to be introduced. This coordinate system takes the top-left corner of each sensor's output image as its origin, and assumes that the physical pixel sizes of the sensors in the horizontal and vertical directions are d and v, respectively. x and d y For the i-th sensor, assume that the physical coordinates of its top-left corner in the image physical coordinate system are (T... x,i ,T y,i Then the ideal principal point projection is located in its pixel coordinate system (u) 0,i ,v 0,i This will generate a specific bias, located in the sensor pixel coordinate system at:

[0097] , ;

[0098] In the formula, u 0,i and v 0,i T represents the principal point coordinates of the i-th sensor in the pixel coordinate system; x,i and T y,i These represent the horizontal and vertical offsets of the upper left corner of the sensor in the image's physical coordinate system, respectively; d x and d y These represent the physical dimensions of a pixel in two directions, respectively.

[0099] Furthermore, define the normalized focal length: , ;

[0100] In the formula, f x f y This represents the normalized focal length, which is the result of normalizing the focal length according to the horizontal and vertical pixel sizes respectively.

[0101] Therefore, the coordinate transformation relationship of the i-th sensor is defined as follows:

[0102] ;

[0103] In the formula, [u i ,v i ,1] T Represents the homogeneous pixel coordinates of a point in the i-th sensor; [x, y, 1] T The u in the matrix represents the corresponding physical coordinates of the image; 0,i and v 0,i Main point; d x and d y This refers to the pixel size.

[0104] Combining the above transformation steps with the perspective projection and discretization steps, the final imaging mapping model of the i-th sensor can be derived, and the intrinsic parameter matrix of the i-th sensor can be constructed:

[0105] ;

[0106] The principal point coordinates (u) 0,i ,v 0,i The location is determined by the physical arrangement of the sensors.

[0107] , ;

[0108] In the formula, u 0,ref and v 0,ref D represents the principal point coordinates of the center of the reference sensor or standard image; x,i D y,i This represents the lateral and longitudinal arrangement deviation of the i-th sensor relative to the reference position on the focal plane.

[0109] In the formula (u 0,ref ,v 0,ref (i) represents the standard image center. Based on this model, the final imaging formula for the i-th sensor is:

[0110] ;

[0111] In the formula, s represents the scaling factor; P w Represents the world coordinates of a point in space; R and t are unified extrinsic parameters; K i Let be the intrinsic parameter matrix of the i-th sensor.

[0112] In a preferred embodiment of the present invention, the two-stage joint calibration specifically includes:

[0113] Phase 1: The initial values ​​of the intrinsic parameters of each sub-sensor are solved by minimizing the single-view reprojection error of the corner features in each sub-image as the optimization objective.

[0114] The second stage: Based on the initial values, the projection consistency constraint of the overlapping field of view between adjacent sub-sensors is introduced, and together with the minimization of the single field of view reprojection error, it is used as the optimization objective to jointly fine-tune the intrinsic parameters and structural parameters.

[0115] In this embodiment, in terms of optimizing target construction, the present invention does not merely minimize the reprojection error of a single sensor, nor does it rely directly on texture similarity for stitching in the pixel domain. Instead, it introduces projection consistency constraints between overlapping fields of view while maintaining the fidelity of single-view imaging. This constraint is established by backprojecting corresponding features in adjacent fields of view onto the same reference physical plane and comparing their physical position differences, thereby ensuring that the optimization result simultaneously considers local imaging accuracy, continuity of overlapping boundaries, and overall geometric uniformity.

[0116] To accurately solve for the intrinsic parameter matrices of each sensor and the structural constants of the array based on parameter decoupling, this study constructs a joint variational optimization model that integrates monocular imaging fidelity and binocular geometric consistency. This model aims to find a set of optimal parameter space vectors. K i Let be the intrinsic parameter matrix of the i-th sensor. Considering the potential non-convex conflict between monocular and binocular constraints in the early stages of optimization, a total objective function with adaptive weighting factors is designed. By dynamically balancing the monocular reprojection residual term and the overlapping field of view consistency residual term, a gradual convergence from the initial global parameter values ​​to the local fine alignment is achieved.

[0117] First, define the monocular reprojection residual term E. reproj To ensure that the optimized intrinsic parameter estimates conform to the pinhole imaging law, for the corner point of the ChArUco calibration plate observed by the i-th sensor in the j-th frame, its physical coordinates in the world coordinate system are denoted as X. p The observed pixel coordinates on the image plane are u i,j,p Reprojection is performed based on the aforementioned common-mirror structure constraint. The reprojection error is defined as the sum of the squared Euclidean distances between all valid observation points and their theoretical projection points:

[0118] ;

[0119] In the formula, Represents the single-field reprojection residual term; Θ represents the set of parameters to be optimized; N represents the number of sensors; M represents the number of calibration image frames; Ω i,j u represents the set of valid observation points in the j-th frame of the i-th sensor; i,j,p Indicates the coordinates of the observed pixel; This represents the theoretical projection position calculated using the current parameters; ρ(·) represents the robust kernel function.

[0120] ρ h (r) represents the Huber kernel function, used to suppress the influence of detection noise or outliers on optimization convergence, and its definition is as follows:

[0121] ;

[0122] In the formula, r represents the residual; δ represents the segmentation threshold of the Huber kernel. A quadratic penalty is applied when |r| is small, and a linear penalty is applied when |r| is large.

[0123] This function combines the least squares method with absolute value error, and is suitable for situations where the residual is small ( In the region of ), it behaves as a quadratic function, guaranteeing differentiability at the zero point and rapid convergence of the optimization process; while in the region of large residuals ( In the region of ), it degenerates into a linear function, thus avoiding the excessive penalty weight imposed on large residuals by traditional squared errors, and significantly improving the robustness of the system under complex lighting or occlusion interference.

[0124] In a preferred embodiment of the present invention, the projection consistency constraint of the overlapping field of view is constructed in the following manner:

[0125] Based on the identity index of the composite calibration plate, determine the pairs of feature points with the same name in the overlapping area of ​​adjacent sub-sensors;

[0126] The corresponding feature point pairs are back-projected onto the same reference physical plane where the composite calibration plate is located through the intrinsic parameters of their respective sub-sensors;

[0127] Constraints are constructed based on the geometric positional differences of the two points on the reference physical plane after back projection.

[0128] In this embodiment, for the overlapping field of view region, this study further introduces an overlapping field of view projection consistency constraint term. Utilizing the topological relationship between viewpoints, the algorithm locks down a set of pairs of feature points with the same name within the overlapping region of adjacent sensors i and j. The core idea of ​​this constraint is that, under ideal geometric correction, pixels with the same name are back-projected onto the physical reference plane (i.e., the ChArUco plane Z=Z) via their respective intrinsic parameters. ref Afterwards, their physical coordinates should coincide. First, define the inverse projection mapping operator. This represents a linear transformation from the pixel coordinate system to the physical plane:

[0129] ;

[0130] In the formula, P plane (K,u) or This means backprojecting pixel u onto the reference plane Z=Z. ref The physical position on the screen; K is the camera intrinsic parameter matrix; u is the homogeneous vector of pixel coordinates; Z ref The reference plane depth.

[0131] Based on this operator, the optical path geometric consistency residual... Formalized as the Euclidean distance deviation between two sets of back-projected points in physical space:

[0132] ;

[0133] In the formula, This represents the consistency residual of the overlapping field of view projection; T represents the set of adjacent field of view pairs; Let represent the set of corresponding feature points in the i-th and l-th fields of view; Φ(·) is the back projection operator.

[0134] This directly penalizes the geometric misalignment of adjacent fields of view on the physical stitching plane, forcing the intrinsic parameter matrix to satisfy monocular imaging while also taking into account the edge alignment of multiple fields of view.

[0135] To mitigate the mutual interference between different constraints in the initial optimization phase, this invention employs a two-stage optimization strategy from coarse to fine: The first stage prioritizes stabilizing the intrinsic parameters of each field of view based on reprojection errors, establishing physically meaningful initial values; the second stage introduces overlapping field-of-view consistency constraints to jointly fine-tune the intrinsic parameters and structural parameters. This phased approach avoids the interference of splicing constraints with non-physical focal length or principal point adjustments before the initial values ​​are stable, thereby improving the interpretability and engineering usability of the results.

[0136] However, directly introducing overlap constraints in the early stages of iteration can lead to severe parameter drift in the intrinsic parameter calculation. The root cause of this deviation lies in the strong coupling of the parameter space in the early optimization phase. Before the initial intrinsic parameter values ​​converge to near their true physical values, the focal length and principal point have extremely high degrees of freedom. To quickly minimize pixel differences in the overlapping region, the optimizer tends to sacrifice the physical realism of the intrinsic parameters by non-physically adjusting the focal length and principal point for scaling and translation, forcibly compensating for physical installation errors between sensors. While this method numerically reduces the stitching residual, it can lead to overfitting. The resulting intrinsic parameters no longer represent the true optical properties, causing the calibration results to be highly susceptible to the initial values ​​and easily trapped in erroneous local extrema.

[0137] To address the aforementioned issues, this study proposes a two-stage joint optimization strategy from coarse to fine, aiming to ensure the physical accuracy of the solution by decoupling the intrinsic parameter estimation and geometric alignment processes. The first stage employs an independent calibration mechanism, setting the overlap term weight to zero and driving the optimization process solely using ChArUco corner reprojection errors. This stage aims to eliminate geometric constraint interference, allowing the algorithm to focus on the accurate calculation of single-channel optical properties, guiding the parameters to converge rapidly to the physical truth space, thus constructing a robust initial value benchmark for subsequent optimizations. The second stage, building upon this foundation, introduces overlap consistency constraints, using the convergence result of the first stage as a warm-up condition, and jointly fine-tunes the parameters using common-view geometric information. The overall optimization objective function is defined as:

[0138] ;

[0139] In the formula, Θ* represents the optimal parameters after joint optimization; λ(t) represents the constraint weights that change with the iteration stage. Let λ(t) represent the overlapping field-of-view consistency term. In the first stage, let λ(t) = 0 to optimize only the reprojection error; in the second stage, gradually increase λ(t) to introduce multi-field-of-view consistency constraints.

[0140] This strategy, while ensuring the stability of the solution, uses a high-weight penalty term in the later stage to finely adjust the intrinsic and structural parameters, and finally obtains a set of optimal geometric parameters that have both physical fidelity and achieve seamless splicing.

[0141] As a preferred embodiment of the present invention, determining the relative position of each sub-image in the global stitching canvas based on the principal point parameter specifically includes:

[0142] Construct a virtual physical plane with the system's optical axis as the origin;

[0143] For each sub-image, calculate the offset vector of its image center relative to its principal point;

[0144] Align the principal points of all sub-images to the origin of the virtual physical plane, and determine their relative positions in the virtual physical plane based on the offset vector of each sub-image;

[0145] The boundary range of the global mosaic canvas is determined based on the relative positions of all sub-images in the virtual physical plane.

[0146] In this embodiment, after joint calibration, the present invention further utilizes the principal point parameters of each sensor and the image center offset relationship to analyze and recover the spatial arrangement of each sub-field of view relative to the system's principal optical axis, and accordingly determines the translation position and boundary range of each sub-image in a unified panoramic canvas. This process is directly driven by the geometric parameters obtained from calibration, without relying on subsequent texture search, feature matching, or empirical stitching amount, and is therefore particularly suitable for multi-field stitching scenarios under conditions of low texture, small overlap, or repetitive texture.

[0147] The high-confidence intrinsic parameter matrix K for each sub-field of view is obtained through joint optimization. i Subsequently, to achieve seamless stitching of multi-field images, the core task is to transform the discrete arrangement of sensors on the physical focal plane into pixel translation vectors in the image domain. Given that this system employs a strict single-lens common optical center structure, with each discrete sensor sharing the same physical principal optical axis, the point where the system's optical axis lands on the calibration plate plane (or any virtual physical plane perpendicular to the optical axis) is a unique, fixed, absolute physical reference point P. ref Based on this physical fact, the principal point (u) on each sensor image... 0,i ,v0,i In fact, it is the same physical point P. ref Projection in different sensor coordinate systems.

[0148] To rigorously establish this correspondence mathematically, it is necessary to analyze the mapping mechanism of the intrinsic parameter matrix to the spatial arrangement: for the A sensor, whose intrinsic parameter matrix K i Define the normalized physical coordinates [x, y, 1] T To pixel coordinates [u,v,1] T Affine mapping relationship:

[0149] ;

[0150] In the formula, [u,v,1] T This represents a homogeneous vector of pixel coordinates; [x, y, 1] T K represents the normalized physical plane coordinates. i Let be the intrinsic parameter matrix of the i-th sensor. This formula shows that the principal point parameters essentially encode the deviation of the sub-field of view relative to the system's principal optical axis.

[0151] As can be seen from the above equation, the principal point parameters in the intrinsic parameter matrix are not only the calibration results of the imaging model, but also encode the first... The absolute geometric anchor points of each sensor relative to the system's optical axis, if all sub-field-of-view images are projected back into physical space, their principal points will inevitably coincide at a single point. Therefore, this study reconstructs the spatial topology of each field of view by projecting the principal points. The specific steps are as follows: First, a virtual panoramic physical plane is constructed with the system's optical axis as the origin (0,0). For the i-th sensor, its image geometric center O... img,i That is, the center of resolution Relative to its principal point O p,p,i (u 0,i ,v 0,i There exists a fixed pixel offset vector d. i :

[0152] ;

[0153] In the formula, d i u represents the offset vector of the center of the i-th sensor image relative to the principal point; img v img This indicates the coordinates of the center of the sensor image; u pp v pp W represents the coordinates of the principal point. i H i Let represent the width and height of the i-th subgraph, respectively.

[0154] d iIt accurately characterizes the direction and distance of the physical installation position of the i-th sensor from the optical axis of the system.

[0155] To restore the true physical arrangement of each viewpoint, a "relative alignment" operation is performed: the principal points of all sub-images are forcibly aligned to the origin (0,0) of the virtual plane. At this point, the... The relative coordinates of any pixel p in the image in the virtual plane It can be represented as:

[0156] ;

[0157] In the formula, This represents the relative coordinates of pixel p in the sub-image within the unified virtual plane.

[0158] Through the above operations, each sub-field of view image will automatically form a staggered overall field of view around the optical axis center based on its actual physical arrangement. At this time, the coordinates of some regions may be negative. In order to construct a valid panoramic canvas, the algorithm further traverses the boundary coordinates of all N fields of view in the virtual plane and calculates their maximum geometric envelope range in the horizontal and vertical directions:

[0159] ;

[0160] In the formula, U min U max V min V max W represents the minimum and maximum boundaries of all sub-fields of view in the horizontal and vertical directions within the unified plane, respectively. i H i This represents the size of the i-th subgraph.

[0161] Based on this envelope boundary, the final resolution W of the panoramic canvas global ×H global and coordinate system translation compensation vector T offset The following is confirmed:

[0162] ;

[0163] ;

[0164] ;

[0165] In the formula, W global and H global These represent the width and height of the panoramic canvas, respectively; T offset This represents the global translation compensation vector introduced to eliminate negative coordinates.

[0166] Ultimately, the absolute translation position of the i-th sub-image in the panoramic canvas is the superposition of its relative displacement and the global compensation vector. This strategy cleverly avoids complex reprojection interpolation calculations, directly utilizes calibration intrinsic parameters to achieve a precise mapping from discrete physical arrangement to a unified panoramic pixel domain, and automatically establishes the panoramic imaging boundary that maximizes the field of view.

[0167] The original paper constructs four sub-images with a certain overlap ratio to simulate the multi-field observation conditions of a single-lens multi-sensor system, in order to verify the effectiveness of the proposed joint calibration method in eliminating geometric conflicts and improving stitching quality. The significance of this experiment lies in comparing the differences in overall stitching effect between two strategies: "considering only the reprojection accuracy of a single field of view" and "considering both the imaging fidelity of a single field of view and the geometric consistency of the overlapping fields of view," rather than simply comparing the magnitude of individual residuals.

[0168] Regarding evaluation metrics, the original paper provides a comprehensive analysis from multiple perspectives, including focal length consistency, principal point projection consistency, relative displacement deviation, total reprojection error, and geometric consistency residuals. This metric system serves to characterize scale uniformity, principal axis reference consistency, spatial topology restoration accuracy, and the constraint balance between single and multiple fields of view. Therefore, it better illustrates that this invention does not simply pursue the minimum of any single residual, but rather seeks optimal overall geometric consistency for splicing applications.

[0169] When evaluating the quality of parameter decoupling, this paper quantitatively assesses the calibration effect from two dimensions: parameter consistency trend and residual convergence characteristics. The core purpose of the simulation experiment is not to pursue absolute regression of parameters to theoretical true values, but to verify the effectiveness of the joint optimization algorithm in eliminating geometric conflicts and improving stitching quality through a controlled environment. In multi-sensor imaging systems, logical consistency among the parameters of each field of view is a prerequisite for achieving seamless stitching. If the focal lengths calculated independently for each field of view are not uniform, it will directly lead to scaling mismatch in the overlapping area. Therefore, this paper introduces a focal length consistency standard deviation. To measure the scale compatibility between different fields of view, it is defined as the calculated focal length relative to the system's average focal length. Standard deviation:

[0170] ;

[0171] In the formula, f represents the standard deviation of focal length consistency. k This represents the focal length calculated for the k-th subfield of view; represents the average focal length of all sub-fields of view; K represents the number of sub-fields of view included in the statistics.

[0172] Meanwhile, deviations in the principal point projection inevitably lead to rotational geometric misalignments between fields of view that cannot be eliminated by simple translation. To address this phenomenon, this paper defines the principal point projection consistency residual. By calculating the principal point c of each sub-sensor k Compensating for physical slice displacement After being mapped onto the physical imaging plane, relative to its group geometric center P centroid Characterized by the degree of discreteness:

[0173] ;

[0174] In the formula, Indicates the principal point projection consistency residual; c k Denotes the principal point of the k-th subfield; Proj(c k ,K k This represents the projection result of mapping the principal point onto the physical plane; Indicates slice coordinate compensation; P centroid This represents the geometric center of each principal point after projection.

[0175] Furthermore, to evaluate the algorithm's ability to recover the sensor spatial topology, this paper introduces a relative displacement deviation ΔT. This reflects the Euclidean distance deviation between the relative displacement vectors between sub-sensors calculated during calibration and the true values ​​of the preset slice coordinates, and is used to characterize the accuracy of the placement of each field of view on the global canvas.

[0176] ;

[0177] In the formula, ΔT represents the relative displacement deviation; T calc T represents the relative displacement recovered based on the calibration results. true This represents the actual relative displacement corresponding to the preset slice coordinates.

[0178] Figure 3 From left to right and from top to bottom, the children are respectively Figure 1 ,son Figure 2 ,son Figure 3 Kazuko Figure 4 , Figure 4 and Figure 5 It is for Figure 3 The stitching effect achieved by combining multiple sub-images. Figure 4 To optimize the effect using only reprojection error, Figure 5 To incorporate the effect of geometric optical path consistency residual optimization, the intrinsic parameters corresponding to the stitching process are shown in Table 1, and the quantitative evaluation indicators are summarized in Table 2. The effect of using only reprojection error optimization is as follows: Figure 4 As shown, even with extremely small single-projection reprojection residuals, significant ghosting and misalignment still occur in the overlapping areas due to geometric conflicts between field-of-view parameters. The effect of introducing geometric optical path consistency residual optimization is as follows: Figure 5 As shown, under this constraint, the parameter logic is unified, and the overlapping areas achieve seamless sub-pixel level connection.

[0179] Table 1 Internal reference calibration results

[0180]

[0181] Table 2 Comparison of quantitative indicators at each stage of joint optimization with multi-objective constraints

[0182]

[0183] As can be seen from the experimental data in Table 2, after joint optimization, focal length consistency, principal point projection consistency, and relative displacement recovery accuracy are all significantly improved. In the stage where only reprojection error is used as the optimization target, although the single-camera reprojection residual is small, there are still significant geometric conflicts between multiple fields of view. After introducing geometric optical path consistency constraints, the total reprojection error only changes slightly, while the alignment between multiple fields of view is significantly improved. This indicates that by introducing overlapping field-of-view projection consistency constraints during the calibration stage, this invention can directly improve the overall stitching quality without excessively sacrificing the imaging accuracy of a single field of view.

[0184] Compared with the traditional methods of "independent calibration + back-end texture stitching" or "independent calibration + empirical translation stitching", this invention can establish a unified geometric relationship during the calibration stage under the condition that the field of view of a single sensor is limited and the overlap between adjacent fields of view is small. In the stitching stage, the calibration parameters are directly used to restore the true arrangement of the sub-fields of view, thereby improving the continuity of multi-field stitching and the consistency of the overall calibration results.

[0185] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for joint calibration and stitching of multiple sensors in a single lens based on co-pose constraints, characterized in that, The method includes: Acquire images from multiple sub-sensors, each containing a local area of ​​the composite calibration plate; Extract corner features from each sub-image and identify the identity index of the corner features on the composite calibration board; A unified imaging geometry model is established, which describes the multiple sub-sensors as coupled imaging units that share the same external pose and differ only in the arrangement of the focal plane. Based on the unified imaging geometry model, a two-stage joint calibration is performed to solve the structural parameters of each sub-sensor that are involved in characterizing the differences in their focal plane arrangement. The optimization objectives of the two-stage joint calibration include minimizing the single-view reprojection error of each sub-image and the projection consistency constraint of the overlapping fields of view between adjacent sub-sensors. Based on the principal point parameters obtained from the solution, the relative positions of each sub-image in the global stitching canvas are determined, and the sub-images are stitched together into a panoramic image accordingly.

2. The single-lens multi-sensor joint calibration and stitching method based on co-pose constraints according to claim 1, characterized in that, The composite calibration plate is the ChArUco calibration plate.

3. The single-lens multi-sensor joint calibration and stitching method based on co-pose constraints according to claim 2, characterized in that, The process of extracting corner features from each sub-image and identifying their identity index includes: Detect ArUco markers in sub-images and decode them to obtain their identity index; Based on the geometric prior of the ArUco markers, interpolation predicts the initial positions of the checkerboard corner points in the sub-image; Sub-pixel refinement is performed in the neighborhood of the initial position to obtain the final coordinates of the chessboard corner points.

4. The single-lens multi-sensor joint calibration and stitching method based on co-pose constraints according to claim 1, characterized in that, The two-stage joint calibration specifically includes: Phase 1: The initial values ​​of the intrinsic parameters of each sub-sensor are solved by minimizing the single-view reprojection error of the corner features in each sub-image as the optimization objective. The second stage: Based on the initial values, the projection consistency constraint of the overlapping field of view between adjacent sub-sensors is introduced, and together with the minimization of the single field of view reprojection error, it is used as the optimization objective to jointly fine-tune the intrinsic parameters and structural parameters.

5. The single-lens multi-sensor joint calibration and stitching method based on co-pose constraints according to claim 1, characterized in that, The projection consistency constraints of the overlapping fields of view are constructed in the following way: Based on the identity index of the composite calibration plate, determine the pairs of feature points with the same name in the overlapping area of ​​adjacent sub-sensors; The corresponding feature point pairs are back-projected onto the same reference physical plane where the composite calibration plate is located through the intrinsic parameters of their respective sub-sensors; Constraints are constructed based on the geometric positional differences of the two points on the reference physical plane after back projection.

6. The single-lens multi-sensor joint calibration and stitching method based on co-pose constraints according to claim 1, characterized in that, The unified imaging geometry model is as follows: The multiple sub-sensors share the same rotation matrix and translation vector from the world coordinate system to the camera coordinate system; The intrinsic parameter matrix of each sub-sensor contains the coordinates of the principal point used to characterize the deviation of its arrangement position on the unified physical focal plane.

7. The single-lens multi-sensor joint calibration and stitching method based on co-pose constraints according to claim 1, characterized in that, The relative positions of each sub-image within the global stitching canvas are determined based on the principal point parameters, specifically including: Construct a virtual physical plane with the system's optical axis as the origin; For each sub-image, calculate the offset vector of its image center relative to its principal point; Align the principal points of all sub-images to the origin of the virtual physical plane, and determine their relative positions in the virtual physical plane based on the offset vector of each sub-image; The boundary range of the global mosaic canvas is determined based on the relative positions of all sub-images in the virtual physical plane.

Images