Calibration method, device and storage medium for dual fisheye panoramic camera

Abstract

The application discloses a calibration method and device of a double fisheye panoramic camera and a storage medium, relates to the technical field of image processing, and comprises the following steps: generating feature descriptors corresponding to real scene natural feature points according to two fisheye images obtained by the double fisheye panoramic camera; performing cross-lens feature matching on the feature descriptors according to a nearest neighbor matching algorithm to obtain a matching result; determining a parameter operation strategy corresponding to the matching result according to a matching result and a binocular imaging parameter calculation rule, and constructing a re-projection error model in combination with a position corresponding relationship of the matching point pairs; determining a parameter optimization convergence target through the re-projection error model, and solving camera calibration parameters in combination with a nonlinear optimization algorithm. The application extracts real scene natural feature points in the overlapping field of view of the two fisheye images, determines a parameter operation strategy through matching and screening, constructs a re-projection error model for optimization and solving, solves the problem of poor calibration effect, and improves calibration precision and scene adaptability.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a calibration method, apparatus and storage medium for a dual fisheye panoramic camera. Background Technology

[0002] In the calibration, imaging, mass production, and maintenance scenarios of dual fisheye panoramic cameras, the ability to accurately calibrate the camera, adaptively correct parameters, and automate maintenance directly affects the panoramic imaging accuracy, equipment mass production efficiency, and long-term operational stability.

[0003] In related technologies, calibration and image correction of dual fisheye cameras are completed by collecting calibration images during the equipment production stage and combining manual shooting with offline calculation to solidify parameters. This method relies on offline manual intervention to complete the calibration data collection and fixed parameter solidification, which is difficult to adapt to the complex application scenarios of mass production and long-term operating condition changes, resulting in poor calibration results.

[0004] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention

[0005] The main objective of this application is to provide a calibration method, device, and storage medium for a dual fisheye panoramic camera, aiming to solve the technical problem of poor calibration results.

[0006] To achieve the above objectives, this application proposes a calibration method for a dual fisheye panoramic camera, the method comprising: After obtaining two fisheye images from a dual fisheye panoramic camera, real-scene natural feature points in the overlapping field of view of the two fisheye images are extracted, and feature descriptors corresponding to the real-scene natural feature points are generated. The feature descriptors are matched across shots using the nearest neighbor matching algorithm, and valid matching point pairs are selected to obtain the matching results. Based on the matching results and the binocular imaging parameter calculation rules, determine the parameter calculation strategy corresponding to the matching results; Based on the parameter calculation strategy, and combined with the positional correspondence of the matching point pairs, a reprojection error model is constructed. The convergence target for parameter optimization is determined by the reprojection error model, and the camera calibration parameters are obtained by combining the nonlinear optimization algorithm.

[0007] In one embodiment, two original fisheye images are obtained through the dual fisheye panoramic camera, and the two original fisheye images are subjected to image standardization processing to obtain the two fisheye images. Based on the initial installation pose parameters of the dual lenses and the radial distortion distribution law of the fisheye lens, an effective overlapping field of view region is defined in the two fisheye images, and an effective image block of the overlapping region is obtained by cropping. Within the effective image block of the overlapping region, the real-scene natural feature points and their corresponding pixel coordinate information are detected to obtain a set of feature point coordinates. Extract the grayscale distribution and gradient change information within a preset neighborhood of the feature points in the feature point coordinate set, and generate the feature descriptor corresponding to the real-scene natural feature point.

[0008] In one embodiment, the feature descriptor is searched point by point using the nearest neighbor matching algorithm to match candidate corresponding points for all feature points, thereby obtaining an initial set of candidate matching point pairs; Based on the nearest neighbor distance and second nearest neighbor distance of the feature descriptor, the distance ratio threshold is calculated; The initial candidate matching point pair set is filtered by distance ratio, and low-confidence matching point pairs with similarity discrimination below the distance ratio threshold are removed to obtain a preliminary filtered matching point pair set; The preliminary set of matched point pairs is subjected to binocular geometric consistency verification according to the random sampling consistency algorithm. Mismatched point pairs that do not conform to the spatial correspondence are removed to obtain a set of valid matched point pairs. The number of point pairs in the set of valid matching point pairs is counted, and the number of point pairs is compared with the calibrated start threshold to obtain the matching result.

[0009] In one embodiment, if the matching result does not meet the calibration start threshold, the overlapping field of view search area is expanded or the matching result is accumulated to obtain a set of qualified matching point pairs that meet the calibration start conditions. Based on the set of qualified matching point pairs, the pixel coordinate correspondence and confidence weight of each matching point pair are extracted to obtain a matching point coordinate mapping dataset including the confidence weight. Based on the binocular imaging parameter calculation rules and the matching point coordinate mapping dataset, construct the spatial geometric constraint relationship of the dual lenses and define the range of camera parameters participating in the calculation. Based on the spatial geometric constraints of the dual-lens camera and the camera parameter range, a parameter calculation strategy is determined with the goal of minimizing the weighted projection error.

[0010] In one embodiment, the range of parameters to be solved, the weight allocation rules and the optimization convergence objective in the parameter calculation strategy are analyzed, and the actual observed pixel coordinates of all matching point pairs are combined to generate a basic dataset for modeling. Based on the fisheye imaging geometry principle, the spatial geometric constraints of dual lenses, and each pair of matching points in the modeling dataset, the reprojection calculation logic of pixel coordinates and reprojection coordinates is constructed. For each pair of matching points, a corresponding projection confidence weight is matched to the reprojection calculation logic. The initial calculation model is constructed with the weighted sum of the reprojection deviations of all matching points as the optimization objective. After verifying the parameter constraints and operational boundaries of the initial calculation model and confirming that it meets the requirements of the parameter operation strategy, the reprojection error model is output.

[0011] In one embodiment, the initial values ​​and step size constraints of the nonlinear optimization algorithm are set by the parameter optimization convergence objective, the set of parameters to be optimized, and the matching confidence weights within the reprojection error model, thereby generating the parameter iteration solution configuration; Based on the parameter iterative solution configuration and the nonlinear optimization algorithm, the camera intrinsic parameters, distortion parameters and dual-lens relative extrinsic parameters are jointly iteratively calculated to obtain the global weighted projection error value. The global weighted projection error value output in each iteration is compared with a preset convergence threshold. When the global weighted projection error value is lower than the preset convergence threshold for a preset number of consecutive iterations and the fluctuation amplitude is less than the set precision, the current optimal parameter set is output. The optimal parameter set is subjected to imaging geometry consistency verification to obtain the camera calibration parameters in a complete standard format.

[0012] In one embodiment, in response to the parameter correction command, real-world natural feature points in the overlapping field of view of the two fisheye images are extracted, and the reprojection correspondence of feature points is calculated according to the calibration parameter correction rules to obtain a set of projection matching point pairs; Based on the set of projection matching point pairs, and combined with the currently effective camera calibration parameters, reprojection calculations are performed point by point to generate the average reprojection error, the average offset of matching points, and the continuity of the stitching seam. The average reprojection error, the average offset of the matching points, and the continuity of the splicing seam are compared with the corresponding index thresholds one by one, and the error value change trajectory of multiple consecutive frames is traced to obtain the index detection results. When any of the detected indicators in the indicator detection results exceeds the indicator threshold, or when a continuous increasing trend of error is detected in multiple consecutive frames, it is determined that the camera parameters have drifted, and the parameter drift judgment result is output.

[0013] In one embodiment, an incremental parameter correction strategy is initiated based on the parameter drift judgment result. The camera calibration parameters are used as the initial values ​​for iteration, and the distortion parameters involved in the camera are kept unchanged. Matching point pairs of consecutive frames are accumulated through a sliding time window mechanism to generate a correction-specific matching point dataset. Based on the aforementioned modified dedicated matching point dataset, with the relative extrinsic parameters of the two lenses as optimization variables, the rotation increment and translation increment of the extrinsic parameters are calculated according to the least squares optimization algorithm to obtain the extrinsic parameter increment correction amount; The relative extrinsic parameters of the dual lenses are updated according to the extrinsic parameter increment correction amount, and the updated target camera calibration parameters are loaded into the panoramic stitching processing module for subsequent image stitching operations.

[0014] In addition, to achieve the above objectives, this application also proposes a calibration device for a dual fisheye panoramic camera, the calibration device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the calibration method for the dual fisheye panoramic camera as described above.

[0015] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the calibration method for the dual fisheye panoramic camera as described above.

[0016] This application provides a calibration method for a dual fisheye panoramic camera, including acquiring two fisheye images from the dual fisheye panoramic camera, extracting real-world natural feature points in the overlapping field of view of the two images and generating corresponding feature descriptors, performing cross-lens feature matching on the feature descriptors according to the nearest neighbor matching algorithm and filtering valid matching point pairs to obtain matching results, determining the parameter calculation strategy corresponding to the matching results based on the binocular imaging parameter calculation rules, constructing a reprojection error model based on the positional correspondence between the parameter calculation strategy and the matching point pairs, and then determining the parameter optimization convergence target through the reprojection error model and solving it using a nonlinear optimization algorithm. The complete calibration process for camera calibration parameters solves the technical problems of existing dual fisheye panoramic camera calibration technology, which relies on dedicated calibration targets, has cumbersome on-site calibration procedures, and lacks targeted parameter solution strategies and struggles to balance calibration accuracy and operational efficiency in targetless natural feature calibration methods. It improves the scene adaptability and deployment convenience of dual fisheye panoramic camera calibration, while ensuring the joint solution accuracy of camera intrinsic parameters, distortion parameters, and relative extrinsic parameters of the two lenses. It achieves efficient and high-precision automatic calibration in targetless scenarios, providing a stable parameter foundation for the alignment accuracy and image consistency of subsequent panoramic image stitching.

[0017] In summary, this application solves the technical problem of poor calibration results by extracting real-world natural feature points from the overlapping fields of view of two fisheye images, determining parameter calculation strategies through matching and filtering, and constructing a reprojection error model for optimization. This improves calibration accuracy and scene adaptability, and ensures the alignment effect of panoramic stitching. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart illustrating the first embodiment of the calibration method for the dual fisheye panoramic camera of this application; Figure 2 Flowchart for parameter calibration in this application; Figure 3 This is a schematic diagram of the overlapping field of view of the dual fisheye images in this application; Figure 4 This is a schematic diagram of the dual fisheye panoramic camera system structure of this application; Figure 5 This is a flowchart illustrating the seventh embodiment of the calibration method for the dual fisheye panoramic camera of this application; Figure 6 This is a flowchart illustrating the eighth embodiment of the calibration method for the dual fisheye panoramic camera of this application; Figure 7 This is a flowchart of the automatic parameter correction process for this application. Figure 8 This is a schematic diagram of the calibration equipment for the dual fisheye panoramic camera of this application.

[0021] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0022] It should be understood that the specific embodiments described herein are merely illustrative of the technical solutions of this application and are not intended to limit this application.

[0023] In related technologies, calibration and image correction of dual fisheye cameras are completed by collecting calibration images during the equipment production stage and combining manual shooting with offline calculation to solidify parameters. This method relies on offline manual intervention to complete the calibration data collection and fixed parameter solidification, which is difficult to adapt to the complex application scenarios of mass production and long-term operating condition changes, resulting in poor calibration results.

[0024] This application provides a solution: First, after obtaining two fisheye images from a dual fisheye panoramic camera, real-world natural feature points in the overlapping field of view of the two fisheye images are extracted, and feature descriptors corresponding to the real-world natural feature points are generated. Then, cross-lens feature matching is performed on the feature descriptors according to the nearest neighbor matching algorithm to filter valid matching point pairs and obtain matching results. Next, based on the matching results and the binocular imaging parameter calculation rules, the parameter operation strategy corresponding to the matching results is determined. Then, based on the parameter operation strategy and the positional correspondence of the matching point pairs, a reprojection error model is constructed. Finally, the parameter optimization convergence target is determined through the reprojection error model, and the camera calibration parameters are obtained by combining a nonlinear optimization algorithm.

[0025] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device capable of performing the above functions, such as a calibration device for a dual fisheye panoramic camera. The following description uses a calibration device for a dual fisheye panoramic camera as an example to illustrate this embodiment and the subsequent embodiments.

[0026] To better understand the technical solution of this application, a detailed description will be provided below in conjunction with the accompanying drawings and specific implementation methods.

[0027] This application provides a calibration method for a dual fisheye panoramic camera, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the calibration method for the dual fisheye panoramic camera of this application.

[0028] In this embodiment, the calibration method for the dual fisheye panoramic camera includes steps S10 to S50: Step S10: After obtaining two fisheye images from the dual fisheye panoramic camera, extract the real-world natural feature points in the overlapping field of view of the two fisheye images, and generate the feature descriptor corresponding to the real-world natural feature points.

[0029] A dual-fisheye panoramic camera is an imaging device equipped with two symmetrically distributed fisheye lenses for acquiring horizontal panoramic field-of-view images. The overlapping field of view of the two fisheye images is the image area where the fields of view of the two fisheye lenses overlap, forming a binocular spatial correspondence. Real-world natural feature points are naturally existing texture inflection points and contour boundaries in the shooting scene that have stable recognizability. Feature descriptors are unique identifiers that characterize the gray-level distribution and gradient changes around a feature point. For example, building corners, tree contour inflection points, road marking boundaries, and transition points between light and dark textures can all be used as real-world natural feature points.

[0030] In this embodiment, the above-mentioned two-channel fisheye image acquisition and feature extraction process can be triggered in five ways. First, device power-on triggering: after the dual fisheye panoramic camera completes power-on initialization, it automatically starts the initial calibration process, acquiring two channels of fisheye images and performing feature extraction to complete the initial parameter calibration. Second, timed periodic triggering: according to a preset calibration cycle scheduling rule, the system automatically acquires two channels of fisheye images and starts feature extraction at fixed intervals to perform periodic parameter verification and calibration updates. Third, user-initiated triggering: the user initiates a calibration request through the device's interactive interface; upon receiving the request, the system acquires two channels of fisheye images and starts feature extraction to perform manually specified calibration operations. Fourth, stitching quality triggering: when the system detects that the misalignment of the seams in the panoramic stitched image exceeds the warning threshold, it automatically acquires two channels of fisheye images and starts feature extraction to trigger targeted parameter calibration correction. Fifth, device restart triggering: after the device completes an abnormal restart or parameter reset, it automatically acquires two channels of fisheye images and starts feature extraction to quickly restore valid calibration parameters.

[0031] Once the calibration process is triggered, the dual fisheye panoramic camera simultaneously acquires original fisheye images of the two fields of view, and then initiates the operation of delineating the overlapping field of view area and extracting real-world natural feature points.

[0032] For example, there are two methods for extracting real-world natural feature points and generating descriptors in the overlapping field of view region. The first is a global uniform detection-based full-data generation. Before the feature extraction process officially starts, the overlapping field of view region is defined based on the initial installation position of the dual lenses. Then, feature point detection is performed pixel by pixel within the entire overlapping region to identify all real-world natural feature points that meet the texture gradient threshold, and the pixel coordinate information of all feature points is recorded. At the same time, for each detected feature point, the gray-level distribution and gradient direction information in its surrounding fixed neighborhood are extracted, and corresponding feature descriptors are generated one by one. After all features are generated, two sets of feature points and descriptors are output uniformly. This method has comprehensive detection coverage, uniform feature point distribution, and is less likely to miss high-value feature points, providing sufficient feature data sources for subsequent matching stages.

[0033] The second approach involves layered generation of distortion weights across different regions. Based on the radial distortion distribution pattern of fisheye lenses, detection weights are assigned to overlapping fields of view, decreasing progressively from the center to the edges. Feature point detection density is increased in the central region with low distortion and stable imaging, while detection density is decreased in the edge regions with high distortion and image distortion. Simultaneously, the overlapping region is divided into high-texture, medium-texture, and low-texture regions according to texture richness. Multi-scale feature point detection is prioritized in high-texture regions, while only strong inflection point feature detection is performed in low-texture regions. Corresponding neighborhood sizes are matched to feature points in different regions to generate feature descriptors adapted to the degree of distortion in each region. Finally, two sets of feature points and descriptors with region weights are aggregated. This method, combined with the distortion characteristics of fisheye imaging, optimizes the detection strategy, effectively reducing feature instability caused by edge distortion, improving the overall matching reliability of feature points, reducing redundant feature computation, and accelerating feature extraction.

[0034] After generating the two sets of feature points and descriptors, the effective range of the overlapping field of view can be defined based on the following method, and the set of effective feature points that can be used for subsequent matching can be selected.

[0035] In one alternative approach, the system first retrieves the pre-stored initial installation pose parameters of the dual lenses. Combined with the field-of-view parameters of the fisheye lens, the theoretical overlapping field of view is calculated. This theoretical range is then used directly as the effective overlapping area, and all feature points falling within this area are included in the subsequent matching process. This method directly delineates the area based on preset parameters, is logically intuitive, computationally efficient, and fast, making it suitable for conventional calibration scenarios with stable installation poses and no parameter drift.

[0036] In another alternative approach, a coarse matching is first performed based on the global grayscale distribution features of the two images to initially estimate the overlap boundary of the two images' fields of view. Then, the boundary is fine-tuned by combining the feature point distribution density near the boundary, eliminating transitional areas with unstable imaging quality at the boundary, ultimately obtaining a precise and effective overlapping field of view region. Simultaneously, a preliminary quality screening is performed on the feature points within the region, eliminating weak feature points with neighborhood texture contrast below a threshold, and retaining high-quality feature points to form an effective feature point set. This method can dynamically adjust the overlapping region range according to the actual imaging effect, is unaffected by initial installation parameter deviations, has stronger adaptability, and can improve the accuracy of subsequent matching.

[0037] In an exemplary scheme for extracting feature points and generating descriptors, the initial mounting pose parameters of the dual lenses and the field of view parameters of the fisheye lens are first loaded to calculate the initial boundary of the theoretical overlapping field of view. Then, grayscale verification of the boundary region is performed, statistically analyzing the texture richness and feature point density within a preset width on both sides of the boundary. Based on the statistical results, the boundary is adjusted inwards, eliminating edge distortion transition zones to obtain the final effective overlapping field of view region. Next, regional feature detection is performed, dividing the effective overlapping region into three levels according to the distortion distribution weight: a central stable region, an intermediate transition region, and an edge distortion region. High-density multi-scale detection is used in the central stable region, medium-density conventional detection in the intermediate transition region, and low-density strong inflection point detection in the edge distortion region. Real-world natural feature points are identified region by region, and pixel coordinates are recorded. Finally, a preliminary feature point quality screening is performed, eliminating weak texture feature points with a neighborhood grayscale variance below a threshold and unstable feature points located at the image edges. Finally, for each retained feature point, the gradient direction histogram information in its corresponding size neighborhood is extracted to generate a standardized feature descriptor. The two sets of effective feature points and descriptors with quality labels are then summarized for subsequent cross-shot feature matching.

[0038] Step S20: Perform cross-camera feature matching on the feature descriptors according to the nearest neighbor matching algorithm, filter out valid matching point pairs, and obtain the matching result.

[0039] The nearest neighbor matching algorithm is a matching algorithm based on feature descriptor similarity calculation, which retrieves the most similar corresponding point for each feature point. Cross-lens feature matching is the process of establishing a correspondence between feature points in two fisheye images. Valid matching point pairs are feature point pairing results that have a real spatial correspondence, verified by similarity and geometric checks. The calibration start threshold is a pre-set minimum number of matching point pairs used to determine whether there is enough data to carry out parameter calibration. The matching result is a comprehensive result output after completing all screening and threshold determination, including the set of valid matching point pairs and the determination result of whether the number of matching point pairs meets the calibration start conditions. The determination result is divided into two categories: meeting the calibration start conditions and not meeting the calibration start conditions.

[0040] Once the effective feature point sets with descriptors from both paths are obtained, the cross-camera feature matching, mismatch filtering, and threshold determination process is initiated, and the final matching result is output.

[0041] For example, there are two ways to implement cross-lens feature matching and effective point pair selection. The first is a combination of unidirectional nearest neighbor retrieval and ratio selection. After the matching process officially starts, the feature point set of the left-side fisheye image is used as a benchmark, and the feature descriptors of each feature point are extracted one by one. In the right-side feature point set, the nearest and second nearest neighbor points with the highest similarity are retrieved, and the similarity ratio between the nearest and second nearest neighbors is calculated. When the ratio is lower than a preset threshold, it is determined to be a valid candidate matching pair; otherwise, the feature point is removed. After all unidirectional retrievals are completed, geometric consistency checks are performed on all candidate matching pairs, and mismatched pairs that do not conform to the geometric relationship of binocular imaging are removed, resulting in a set of valid matching point pairs. This method has a simple and stable matching logic, low computational load, and fast execution speed, and is suitable for conventional calibration scenarios with a moderate number of feature points and clear textures.

[0042] The second method combines bidirectional mutual neighbor matching with block constraint filtering. Bidirectional neighbor retrieval is performed using the feature point sets of the left and right paths as benchmarks. Only when the retrieval results in both directions correspond to each other are retained as initial candidate matching pairs. Subsequently, the overlapping field of view is divided into multiple uniformly sized sub-blocks. The average coordinate offset of matching point pairs within each sub-block is calculated, and discrete mismatched pairs with offsets exceeding the sub-block average threshold are removed. Finally, a global geometric consistency check is performed to further remove abnormal matching pairs that do not conform to spatial constraints, resulting in a set of valid matching point pairs. This method significantly reduces the probability of mismatches through bidirectional mutual checking and further filters discrete error points by combining block offset constraints, resulting in a significant improvement in matching accuracy. It is suitable for calibration in complex scenes with complex textures and many interference points.

[0043] Once a set of valid matching point pairs is obtained, the calibration start threshold determination is performed to generate the final matching result.

[0044] In one alternative approach, the total number of valid matching point pairs is directly counted and compared with a preset calibration start threshold. If the total number is greater than or equal to the calibration start threshold, the calibration start condition is met. If the total number is less than the calibration start threshold, the calibration start condition is not met. Finally, the point pair set and the judgment result are encapsulated together as the matching result output. This method has a direct and clear judgment logic, extremely low computational overhead, and fast response speed, making it suitable for fast calibration scenarios with high real-time requirements.

[0045] In another optional approach, in addition to counting the total number of valid matching point pairs, the spatial distribution uniformity of matching points in the overlapping field of view is also counted. Both the total number and the distribution uniformity are used as the judgment criteria. When the total number reaches a threshold and the distribution uniformity meets the minimum requirement, it is determined that the calibration start condition is met. If the total number meets the threshold but the distribution is excessively concentrated, or if the total number does not meet the threshold, it is determined that the calibration start condition is not met. The final output is a matching result containing both dimensions of judgment. This method introduces distribution features to assist in the judgment, which can avoid parameter calculation deviations caused by the concentration of matching points, improve the rationality of the calibration start judgment, and adapt to high-precision calibration scenarios.

[0046] In an exemplary scheme for generating matching results, firstly, two sets of valid feature points with descriptors are loaded, and a bidirectional nearest neighbor retrieval process is initiated. Using the feature points of the left and right paths as benchmarks, the descriptor similarity between each feature point and all feature points of the other path is calculated, retrieving the most similar corresponding point for each feature point. Then, a bidirectional consistency check is performed, retaining only the matching pairs that correspond to each other in the bidirectional retrieval, generating an initial set of candidate matching point pairs. Next, a similarity ratio check is performed, calculating the ratio of the nearest neighbor to the second nearest neighbor for each matching pair, and removing low-discrimination matching pairs with ratios exceeding a preset threshold, resulting in a first-level filtered set of matching point pairs. Then, a block-based local check is performed, dividing the overlapping field of view into uniform sub-blocks, calculating the average coordinate offset of matching point pairs within each sub-block, and removing discrete matching pairs with offsets exceeding three times the average threshold of the sub-block, resulting in a second-level filtered set of matching point pairs. Finally, a global geometric consistency check is performed, estimating the stereo fundamental matrix based on a random sampling consistency algorithm, checking the geometric deviation of all matching pairs, and removing mismatched pairs exceeding the deviation threshold, resulting in a set of valid matching point pairs. Subsequently, the total number of valid matching point pairs and the uniformity of their regional distribution are statistically analyzed. These two indicators are compared one by one with the calibration start threshold and uniformity requirements. If both indicators meet the requirements, a judgment result that meets the calibration start conditions is generated; if either one does not meet the requirements, a judgment result that does not meet the calibration start conditions is generated. Finally, the set of valid matching point pairs and the judgment result are packaged into a complete matching result output. If the judgment is not met, a matching data supplementation mechanism is triggered synchronously, automatically expanding the overlapping field of view search range or accumulating the matching results of multiple consecutive frames to supplement the data until the judgment is met, after which the subsequent parameter calculation stage begins.

[0047] It should be noted that when the matching result is determined to be inconsistent with the calibration start condition, the system automatically executes the multi-frame cumulative supplementation mechanism, accumulating valid matching point pairs frame by frame and updating the number statistics in real time until the calibration start threshold is reached. If the target is still not met after accumulating the preset number of frames, the system returns to the feature extraction stage to expand the overlapping field of view search range.

[0048] Step S30: Determine the parameter calculation strategy corresponding to the matching result based on the matching result and the binocular imaging parameter calculation rules.

[0049] The binocular imaging parameter calculation rules are standardized calculation guidelines based on the principles of binocular vision geometry, used to solve for the camera's intrinsic and extrinsic parameters. The parameter calculation strategy is a comprehensive execution plan for the parameter solution range, optimization objectives, and iteration rules set for this calibration task. For example, the binocular imaging parameter calculation rules cover core calculation logic such as epipolar geometric constraints, reprojection error calculation, and parameter iterative optimization. Distortion parameters are correction coefficients used to correct for stretching and offset distortion in fisheye lens imaging, such as radial distortion coefficients k1 and k2 and tangential distortion coefficients p1 and p2.

[0050] Once the matching results that meet the calibration start conditions are obtained, the set of valid matching point pairs is extracted, and the start parameter calculation strategy is determined by combining the binocular imaging parameter calculation rules.

[0051] For example, there are two ways to determine the parameter calculation strategy. The first is a preset strategy matching method, which pre-stores multiple sets of standard parameter calculation strategies adapted to different matching scales and accuracy requirements. Each strategy corresponds to the matching point number range, distribution uniformity, and scene type adaptation conditions. The number of point pairs, distribution uniformity, and average confidence of the current matching result are compared with the adaptation conditions of each strategy, and the most suitable standard calculation strategy is directly matched and selected as the parameter calculation strategy for this calibration. This method relies on direct matching based on preset strategies, has simple logic, fast response speed, and the strategy has been verified to have high stability in a large number of scenarios, making it suitable for regular periodic calibration and rapid calibration scenarios.

[0052] The second method is dynamic customized generative determination. It extracts core features from the current matching results, such as the number of point pairs, spatial distribution density, and region confidence weights. Combined with the binocular imaging parameter calculation rules, it dynamically adjusts the category range, initial value interval, weight allocation rules, and convergence accuracy requirements of the parameters to be solved, generating a customized parameter calculation strategy that is fully adapted to the current matching data. Simultaneously, it verifies the feasibility of the generated strategy, confirming that the parameter range and convergence rules conform to the imaging geometric constraints. After verification, the output is the parameter calculation strategy for this calibration. This method allows for flexible strategy customization based on actual matching data, resulting in higher adaptation accuracy. It can fully exploit the effective information in the matching data, improve the accuracy of parameter solving, and is suitable for high-precision calibration and calibration in complex scenarios.

[0053] Once the generation method of the parameter calculation strategy is determined, the range of the parameters to be solved and the optimization objective can be defined based on the following methods.

[0054] In one alternative approach, camera intrinsic parameters, distortion parameters, and relative extrinsic parameters between the two lenses are all included in the solution scope. A unified optimization weight and convergence threshold are set to construct a computational strategy for joint optimization of all parameters. This method provides comprehensive parameter solutions and can complete the calibration of all parameters in one go, making it suitable for initial calibration and full parameter calibration scenarios.

[0055] In another alternative approach, the camera's intrinsic distortion parameters are locked to fixed values, and only the relative extrinsic parameters between the two lenses are included in the solution. Higher optimization weights and more refined convergence thresholds are set for the extrinsic parameters, constructing an incremental optimization strategy for them. This method solves for fewer variables, has a faster iteration speed, and can quickly complete extrinsic parameter drift correction, making it suitable for scenarios involving periodic parameter verification and drift correction.

[0056] In an exemplary scheme with a defined parameter calculation strategy, the set of valid matching point pairs and the binocular imaging parameter calculation rules from the matching results are first loaded. The total number of point pairs, regional distribution uniformity, and average confidence weight of the matching results are then statistically analyzed. Next, a calibration scenario determination is performed. If it is the first calibration or full calibration scenario, all camera intrinsic parameters, distortion parameters, and relative extrinsic parameters of the dual lenses are designated as parameters to be solved, and the initial value range for each parameter is set based on the lens's factory parameters. If it is a periodic verification or drift correction scenario, the camera intrinsic and distortion parameters are locked to their current stored values, and only the relative extrinsic parameters of the dual lenses are designated as parameters to be solved, with a narrow initial value range centered on the current extrinsic parameter value. Then, optimization weight rules are set, assigning corresponding weights based on the regional confidence of the matching points. Matching points in the central stable region have higher weights, while matching points in the edge distortion region have lower weights. Finally, the convergence objective and iteration rules are set. The core convergence objective is to minimize the weighted projection error. The error convergence threshold and maximum number of iterations for the corresponding accuracy level are set. A complete parameter calculation strategy is formed and the output is used to construct the subsequent reprojection error model.

[0057] It should be noted that in some special cases, if the matching points are concentrated in a single region, the initial range of parameter values ​​will be automatically expanded and the convergence speed will be reduced to avoid the solution from getting trapped in a local optimum.

[0058] Step S40: Based on the parameter calculation strategy, construct a reprojection error model by combining the positional correspondence of the matching point pairs.

[0059] The positional correspondence of matching point pairs represents the pixel coordinate mapping of the same spatial point in two fisheye images. The reprojection error model is a mathematical calculation model used to calculate the deviation between the projected coordinates and the actual observed coordinates of a spatial point. For example, the input to the reprojection error model includes camera parameters and the coordinates of the matching points, and the output is the overall projection deviation value.

[0060] Once the parameter calculation strategy and the correspondence between the matching point pairs are obtained, the construction process of the reprojection error model is started.

[0061] For example, there are two ways to construct the reprojection error model. The first is to construct a globally uniform weight model. Based on the parameters to be solved and the fisheye imaging projection logic defined by the parameter calculation strategy, a reprojection calculation link is established, which transforms pixel coordinates through camera intrinsic parameters, distortion parameters, and extrinsic parameters. The same calculation weight is assigned to all matching point pairs, and the arithmetic mean of the reprojection deviations of all matching points is used as the optimization objective to construct a globally uniform weight reprojection error model. This method has a simple model structure, clear calculation logic, and strong iterative solution stability, making it suitable for conventional calibration scenarios where matching points are evenly distributed and scene imaging quality is consistent.

[0062] The second approach involves constructing a region-specific adaptive weighted model. Combining fisheye distortion distribution with the confidence level of matching points, the model divides matching point pairs into three levels: high-confidence, medium-confidence, and low-confidence regions. Calculation weights are assigned to matching points at different levels with progressively decreasing weights. A distortion-sensitive term is introduced, adding distortion correction weight factors to matching points in distorted edge regions to correct the reprojection deviation calculation logic in these areas. Finally, the total weighted projection deviation of all matching points is used as the optimization objective to construct a region-specific adaptive weighted reprojection error model. This approach fully considers the distortion characteristics of fisheye imaging and the quality differences of matching points. By guiding the optimization direction through differentiated weights, it effectively improves the overall accuracy of parameter solving and is suitable for high-precision calibration and complex imaging scenarios.

[0063] Once the main model is constructed, model constraint verification can be performed in the following manner to output a usable error model.

[0064] In one alternative approach, hard boundary constraints are added to the model based on the parameter value range defined by the parameter calculation strategy. This ensures that all parameters remain within a reasonable range during iteration, preventing invalid calculations caused by parameter out-of-bounds errors. This method has simple constraint logic, effectively guarantees the stability of the iteration process, and is suitable for conventional optimization solution scenarios.

[0065] In another alternative approach, in addition to hard boundary constraints on parameter values, soft constraints on the geometric rationality of the binoculars are added. A reasonable fluctuation range is set for the rotation angle and translation distance of the two lenses relative to their extrinsic parameters. When the range is exceeded, a penalty term is applied to increase the error value, guiding the optimization process to converge towards a geometrically reasonable direction. This method further improves the physical rationality of the solution results through soft constraints, avoiding mathematically optimal but geometrically invalid results, and is suitable for high-precision calibration scenarios.

[0066] In an exemplary scheme for constructing a reprojection error model, the parameter calculation strategy and the positional correspondence of matching point pairs are first loaded, and the list of parameters to be solved, the initial value range, and the weight allocation rules are extracted. Then, a reprojection calculation chain is built. Following the fisheye lens imaging projection logic, a complete calculation process is established sequentially, including pixel coordinate distortion correction, intrinsic parameter transformation, extrinsic parameter spatial transformation, and another path of projection imaging, realizing the mapping calculation from a single path of pixel coordinates to another path of reprojection coordinates. Next, weight allocation is performed. Based on the distortion degree and matching confidence of the region where the matching point is located, corresponding calculation weights are assigned to each matching point pair, with the highest weight in the high-confidence region at the center and the lowest weight in the low-confidence region at the edge. Constraints are then added, setting hard boundaries for the values ​​of all parameters to be solved, while setting geometric rationality soft constraints and corresponding penalty coefficients for the relative extrinsic parameters of the two lenses. Finally, the reprojection calculation chain, weight allocation rules, and constraints are integrated to construct a reprojection error model with the objective of minimizing the sum of weighted projection errors. After model validity verification, the model is output for subsequent parameter optimization.

[0067] It should be noted that in some special cases, if the number of matching points is large, a subset model can be constructed using random sampling for pre-iteration to speed up the initial convergence.

[0068] Step S50: Determine the convergence target for parameter optimization through the reprojection error model, and solve for the camera calibration parameters using a nonlinear optimization algorithm.

[0069] The convergence objective of parameter optimization is an error accuracy index used to determine the termination condition during the parameter iterative optimization process. Nonlinear optimization algorithms are numerical computation methods that solve for the minimum value of a nonlinear function through iteration. Camera calibration parameters are a complete set of parameters obtained through calibration, used to describe the camera's imaging characteristics and spatial pose. For example, camera calibration parameters include three core types of parameters: camera intrinsic parameters, distortion parameters, and relative extrinsic parameters between the two lenses.

[0070] Once the reprojection error model is built, the parameter optimization solution process is started, and the final camera calibration parameters are output.

[0071] For example, there are two methods for optimizing camera calibration parameters. The first is a fixed-step global iterative solution. Using the initial value set by the parameter calculation strategy as the starting point, a nonlinear optimization algorithm is employed to perform global iterations with a fixed step size. In each iteration, the values ​​of all parameters to be solved are updated, and the current reprojection error value is calculated simultaneously. When the error value is below the convergence threshold for several consecutive iterations and the fluctuation amplitude is less than the set accuracy, the iteration is terminated, and the current parameters are output as the camera calibration parameters. This method offers a stable and controllable solution process, simple iteration rules, and is less prone to iteration divergence, making it suitable for calibration scenarios with conventional accuracy requirements.

[0072] The second approach is adaptive step-size hierarchical iterative solution. This method divides the optimization process into two stages: coarse convergence and fine convergence. In the coarse convergence stage, a larger iteration step size is used to narrow the parameter value range, allowing the reprojection error to quickly decrease to near the optimal range. In the fine convergence stage, the iteration step size is automatically reduced, and the parameters are finely adjusted while gradually tightening the convergence threshold until the error meets the accuracy requirements. During the iteration process, the error reduction trend is monitored in real time. If the error continues to rise, the parameters are automatically adjusted back and the step size is reduced to ensure stable convergence. The final converged parameters are output as camera calibration parameters. This method balances solution speed and accuracy through hierarchical iteration, and the adaptive step size effectively avoids iterative oscillations, improving solution efficiency and final accuracy. It is suitable for high-precision calibration and multi-parameter joint optimization scenarios.

[0073] After the parameter iteration is completed, parameter verification can be performed in the following way to output the final calibration parameters.

[0074] In one alternative approach, the average reprojection error of the current matched point pair is used as the verification metric. If the error value is lower than a preset acceptable threshold, the parameters are deemed valid and directly output as the final camera calibration parameters. This method features simple verification logic, fast execution speed, and is suitable for common calibration scenarios.

[0075] In another alternative approach, reprojection error verification is first performed using the current matching point. Then, multiple frames of images not involved in the solution are randomly selected for cross-validation. The average reprojection error and the degree of misalignment of the stitching seams are calculated. When all indicators meet the threshold requirements, the parameters are deemed valid and output. This method ensures the generalization of parameters through cross-validation, avoids overfitting, and has higher parameter reliability, making it suitable for high-precision calibration scenarios.

[0076] In an exemplary scheme for solving camera calibration parameters, the reprojection error model and parameter calculation strategy are first loaded, and the parameter optimization convergence objective, initial values ​​of the parameters to be solved, and maximum number of iterations are extracted. Then, a hierarchical iterative solution is initiated, initially entering a coarse convergence phase with a large iteration step size. A nonlinear optimization algorithm is used to perform global parameter iteration, updating all parameters to be solved and calculating the weighted projection error in each round. When the error value decreases to a preset coarse convergence threshold, the process transitions to a fine convergence phase. In the fine convergence phase, the iteration step size is automatically reduced, and refined parameter iteration is performed, simultaneously tightening the error convergence threshold. During the iteration process, the error change trend is monitored in real time; if the error continuously increases, the parameters are automatically reverted and the step size is further reduced. When the error value is below the final convergence threshold for several consecutive rounds and the fluctuation amplitude is less than the accuracy requirement, the iteration terminates, obtaining a preliminary optimal parameter set. Next, parameter validity is verified by calculating the average reprojection error of all matching points and verifying the geometric rationality of the parameter values, confirming that all indicators meet the requirements. Finally, the parameters are organized into a standard format, generating a complete set of camera calibration parameters, which are then output for subsequent panoramic image stitching processing.

[0077] It should be noted that in some special cases, if convergence is not achieved after reaching the maximum number of iterations, the process returns to the feature matching stage to re-select matching point pairs, adjusts the computation strategy, and then executes the solution again.

[0078] Further, please refer to Figure 2 , Figure 2This is a flowchart of the parameter calibration process for this application. The calibration process of the dual fisheye panoramic camera consists of seven steps: image acquisition, image preprocessing, feature extraction, feature matching, camera parameter calculation, parameter optimization, and generation of calibration parameters. First, the dual fisheye camera device simultaneously acquires two fisheye images at a frame rate of 10fps to 60fps. The image resolution can be 1920×1920, 2560×2560, or higher. There is a certain overlap in the field of view between the two fisheye lenses. The system uses the natural scene information in this overlap area to complete the automatic calibration, without the need for dedicated calibration boards such as checkerboard or dot arrays. After acquisition, the two images enter the image preprocessing module, where grayscale processing, noise reduction processing, contrast enhancement, and brightness equalization processing are performed sequentially. The noise reduction processing can use Gaussian filtering, median filtering, or bilateral filtering. The preferred filter window size is 3×3 to 7×7 to improve the stability of subsequent feature detection. After preprocessing, the system proceeds to the feature extraction module. It automatically extracts natural feature points with significant grayscale or texture variations, such as building edges, door and window outlines, road boundaries, streetlights, tree textures, and billboard edges, from the overlapping areas of the dual fisheye images. Feature extraction algorithms can include Oriented Fast and Rotated BRIEF (ORB), Features from Accelerated Segment Test (FAST), Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), or Accelerated KAZE (AKAZE). The Oriented Fast and Rotated BRIEF (ORB) algorithm is preferred. The system extracts 500 to 3000 feature points per image and generates a corresponding feature descriptor for each feature point. The system then enters the feature matching module. First, the K-Nearest Neighbors (KNN) matching algorithm is used to match the feature descriptors in the two fisheye images to obtain candidate matching point pairs. Then, the distance ratio filtering method is used to remove low-confidence matching points. Furthermore, the Random Sample Consensus (RANSAC) algorithm is used to eliminate mismatched points and obtain a stable set of matching points. When the number of valid matching points is greater than 50, automatic calibration calculation is started. When the number of valid matching points is insufficient, the system automatically expands the search area or accumulates the matching results in multiple consecutive frames of images to improve calibration stability.After obtaining stable matching points, the camera parameter calculation module calculates the dual fisheye camera parameters, including intrinsic and extrinsic parameters, based on the matching point pairs. The intrinsic parameters include focal length, principal point coordinates, and distortion parameters, while the extrinsic parameters include rotation matrix and translation vector. The system first establishes binocular geometric constraint relationships based on the matching points and constructs an overall reprojection error model. Then, through the parameter optimization stage, the Levenberg-Marquardt (LM) nonlinear optimization algorithm is used to jointly optimize the intrinsic, extrinsic, and distortion parameters to minimize the overall reprojection error. Finally, the optimal calibration parameters are generated and written to the storage module for use by the panoramic stitching module. During equipment operation, the system continuously performs parameter drift detection, periodically re-extracting natural feature points from the overlapping area of ​​the dual fisheye images, and calculating the reprojection error, average matching distance, and stitching seam continuity index of the current matching point set. When any of the following conditions are met: average reprojection error exceeds a preset threshold, average offset of matching points exceeds a preset threshold, continuous misalignment occurs in the stitching seam area, or error continues to increase over multiple frames, camera parameter drift is determined. The error threshold can be set from 1 to 5 pixels depending on the device resolution. Upon detecting parameter drift, the system automatically initiates a parameter correction process, using the currently calibrated parameters as initial values, and only correcting the rotation matrix and other parameters in the external parameters. The translation vector is incrementally optimized, and a sliding time window mechanism is used to count stable matching points in the most recent multiple frames of images. The incremental correction amount of the extrinsic parameters is calculated and the parameters are updated through the least squares optimization algorithm. The updated parameters are immediately used for subsequent panoramic stitching processing. At the same time, the system sets up a parameter reliability evaluation mechanism. Only when consistent correction results are obtained in multiple consecutive time windows will the new parameters be written to the storage module. Otherwise, the original parameter configuration is maintained. In this way, when the equipment is subjected to temperature changes, mechanical vibration, deformation of the mounting bracket, or slight changes in lens posture due to long-term operation, parameter drift is automatically detected and corrected in real time. This maintains the accuracy and continuity of panoramic stitched images over a long period of time without the need for manual recalibration.

[0079] Second Embodiment This embodiment provides an exemplary scheme for extracting real-world natural feature points and generating descriptors in overlapping field-of-view regions. In this example, image standardization is first performed on the two original fisheye images. Then, the effective overlapping field-of-view region is delineated by combining the initial mounting pose of the dual lenses and the radial distortion law of the fisheye. Subsequently, real-world natural feature points are detected within the region and their pixel coordinates are recorded. Finally, neighborhood features of the feature points are extracted to generate corresponding feature descriptors, thereby obtaining a high-quality set of feature points and descriptors that can be used for subsequent cross-lens matching. Step S10 includes steps A11 to A14: Step A11: Obtain two original fisheye images through the dual fisheye panoramic camera, and perform image standardization processing on the two original fisheye images to obtain the two fisheye images.

[0080] Step A12: Based on the initial installation pose parameters of the dual lenses and the radial distortion distribution law of the fisheye lens, the effective overlapping field of view area is delineated in the two fisheye images, and the effective image block of the overlapping area is cropped.

[0081] Step A13: Within the effective image block of the overlapping region, detect the real-world natural feature points and their corresponding pixel coordinate information to obtain a set of feature point coordinates.

[0082] Step A14: Extract the grayscale distribution and gradient change information of the feature points in the preset neighborhood of the feature points in the feature point coordinate set, and generate the feature descriptor corresponding to the real-scene natural feature point.

[0083] Image standardization is a series of preprocessing operations performed on the original fisheye image to unify its format, suppress noise, and balance image quality. It is a preprocessing step to ensure the stability of subsequent feature detection. The effective overlapping field of view is the image region where the fields of view of the two fisheye lenses overlap and the imaging quality meets the requirements for feature detection. It is the effective range for extracting real-world natural feature points. The effective image block of the overlapping region is an independent image unit cropped from the two fisheye images, corresponding to the effective overlapping field of view. It is the direct processing object for feature point detection. Real-world natural feature points are naturally existing image inflection points and boundary points in the shooting scene that possess stable grayscale and texture characteristics. They are the basic matching units for establishing the spatial correspondence between the two images. Feature descriptors are feature vectors that characterize the grayscale distribution and gradient change patterns in the neighborhood of a feature point. They are the core basis for measuring the similarity of different feature points. For example, image standardization includes grayscale processing, Gaussian filtering for noise reduction, local contrast enhancement, and global brightness equalization; real-world natural feature points include building corners, door and window outline inflection points, road boundary intersections, tree texture bifurcation points, and billboard edge inflection points.

[0084] In this example, when delineating the effective overlapping field of view to obtain the effective image block of the overlapping area, it can be done directly according to the theoretical pose. That is, the pre-stored initial installation pose parameters of the dual lenses and the fisheye field of view parameters are retrieved to calculate the theoretical overlapping field of view boundary, and the effective image block of the overlapping area is obtained by cropping according to the boundary. Alternatively, it can be dynamically delineated by coarse matching of gray-level features. First, global coarse matching of gray-level features is performed on the two images to initially estimate the overlapping field of view boundary. Then, the boundary is adjusted inward by combining the texture richness of the boundary area. After removing the edge distortion area, the effective image block of the overlapping area is obtained by cropping, thus completing the delineation of the effective overlapping field of view.

[0085] After cropping the effective image blocks in the overlapping regions, the real-world natural feature point detection process is initiated. For each effective image block in the overlapping region, real-world natural feature points that meet the texture gradient requirements are detected region by region. The pixel coordinate information of each feature point is recorded, and the coordinates of the feature points are summarized to obtain a set of feature point coordinates. Then, for each feature point in the set of feature point coordinates, the gray-level distribution and gradient direction information in its surrounding preset neighborhood are extracted to generate a unique feature descriptor. Finally, two sets of feature points with descriptors are obtained. In this way, through hierarchical region delineation and hierarchical feature detection, the overall quality and matching reliability of feature points are improved, and subsequent matching errors caused by edge distortion and weak texture features are avoided.

[0086] For example, there are two implementation methods for real-scene natural feature point detection and descriptor generation. The first is a global uniform detection-based full generation. After the feature detection process officially starts, feature point detection is performed pixel by pixel in the entire overlapping region of the effective image block. All real-scene natural feature points that meet the texture gradient threshold are identified, and the pixel coordinate information of all feature points is recorded. At the same time, for each detected feature point, the gray-level distribution and gradient direction information in its surrounding fixed neighborhood are extracted, and corresponding feature descriptors are generated one by one. After all features are generated, two sets of feature points and descriptors are output uniformly. This method has comprehensive detection coverage, uniform feature point distribution, and is less likely to miss high-value feature points, providing sufficient feature data sources for subsequent matching stages.

[0087] The second approach involves layered generation of distortion weights across different regions. Based on the radial distortion distribution pattern of fisheye lenses, detection weights are assigned to effective image blocks in overlapping regions, decreasing progressively from the center to the edge. Feature point detection density is increased in the central region with low distortion and stable imaging, while detection density is decreased in the edge region with high distortion and image distortion. Simultaneously, the overlapping region is divided into high-texture, medium-texture, and low-texture regions according to texture richness. Multi-scale feature point detection is prioritized in high-texture regions, while only strong inflection point feature detection is performed in low-texture regions. For feature points in different regions, corresponding neighborhood sizes are matched to generate feature descriptors adapted to the degree of distortion in each region. Finally, two sets of feature points and descriptors with region weights are aggregated. This method, combined with the distortion characteristics of fisheye imaging, optimizes the detection strategy, effectively reducing feature instability caused by edge distortion, improving the overall matching reliability of feature points, reducing redundant feature computation, and accelerating feature extraction.

[0088] Further, please refer to Figure 3 , Figure 3This is a schematic diagram of the overlapping area of ​​the dual fisheye image fields of view in this application. The dual fisheye panoramic camera is equipped with two fisheye imaging units, fisheye lens 1 and fisheye lens 2. The two fisheye lenses cover their respective shooting field of view areas, and the overlapping part between the two shooting areas constitutes the overlapping feature point area. This overlapping field of view area contains a large number of image feature points with stable grayscale and texture characteristics, which can be used to perform cross-lens feature matching calculations between the two fisheye images, providing a core spatial correspondence basis for subsequent camera calibration parameter solving and panoramic image stitching alignment.

[0089] Third Embodiment This embodiment provides an exemplary scheme for cross-camera feature matching, effective matching point pair screening, and threshold determination. In this example, firstly, a point-by-point similarity retrieval is performed on two feature descriptors using a nearest neighbor matching algorithm to obtain an initial set of candidate matching point pairs. Then, a distance ratio threshold is calculated based on the nearest and second nearest neighbor distances of the feature descriptors to complete the initial screening of low-confidence point pairs. Subsequently, a random sampling consensus algorithm is used to perform binocular geometric consistency verification to eliminate false matching point pairs. Finally, the number of effective matching point pairs is counted and compared with the calibrated start threshold, thereby obtaining the matching result containing the threshold determination conclusion. Step S20 includes steps B11 to B15: Step B11: Perform point-by-point similarity retrieval on the feature descriptors using the nearest neighbor matching algorithm to match candidate corresponding points for all feature points, thereby obtaining an initial set of candidate matching point pairs.

[0090] Step B12: Calculate the distance ratio threshold based on the nearest neighbor distance and the second nearest neighbor distance of the feature descriptor.

[0091] Step B13: Perform distance ratio filtering on the initial candidate matching point pair set, and remove low-confidence matching point pairs in the initial candidate matching point pair set whose similarity discrimination is lower than the distance ratio threshold, to obtain a preliminary filtered matching point pair set.

[0092] Step B14: Perform binocular geometric consistency verification on the preliminary set of matched point pairs according to the random sampling consistency algorithm, remove mismatched point pairs that do not conform to the spatial correspondence, and obtain a set of valid matched point pairs.

[0093] Step B15: Count the number of point pairs in the set of valid matching point pairs, and compare the number of point pairs with the calibrated start threshold to obtain the matching result.

[0094] The initial candidate matching point pair set is a collection of all feature point pairings obtained through nearest neighbor similarity retrieval but without credibility verification. It serves as the foundational data source for subsequent hierarchical screening. The distance ratio threshold is a critical judgment value used to measure the discriminative power of matching point pairs. It is determined by the ratio of the nearest neighbor distance to the second nearest neighbor distance of the feature descriptor and is the core criterion for filtering low-credibility matching pairs. The preliminary screening matching point pair set is the set of matching point pairs remaining after removing low-credibility point pairs through distance ratio screening. It serves as the input for geometric consistency verification. Binocular geometric consistency verification is a verification process based on binocular visual spatial constraints to verify whether matching point pairs conform to the true imaging geometry. It is the core means of eliminating mismatched point pairs.

[0095] The effective matching point pair set is a collection of matching point pairs that have been fully screened and verified and have a real spatial correspondence. It forms the core data foundation for subsequent parameter calculations. The calibration start threshold is a pre-set minimum number of effective matching point pairs used to determine whether there is sufficient data to carry out parameter calibration. The matching result is a comprehensive output after all screening and threshold determination are completed, including a judgment on whether the effective matching point pair set and the number of point pairs meet the calibration start conditions. For example, the initial candidate matching point pair set contains all the one-to-one corresponding feature point pairs retrieved, the distance ratio threshold is usually set to a value in the range of 0.6 to 0.8, and the point pairs in the effective matching point pair set all satisfy both similarity and geometric constraints.

[0096] In this example, when performing point-by-point similarity retrieval on feature descriptors to obtain the initial candidate matching point pair set, a unidirectional benchmark retrieval method can be used. This involves using the feature point set of one fisheye image as a benchmark, extracting feature descriptors one by one, and retrieving the two candidate corresponding points with the highest similarity from the feature point set of the other image. The distance information between the nearest and second nearest neighbors is recorded. After completing the point-by-point retrieval, the results are summarized to obtain the initial candidate matching point pair set. Alternatively, a bidirectional mutual inspection retrieval method can be used. Bidirectional nearest neighbor retrieval is performed using the feature point sets of both fisheye images as benchmarks. Only when the retrieval results in both directions correspond to each other are included in the initial candidate matching point pair set, thus completing the collection of the initial candidate matching point pairs.

[0097] After constructing the initial candidate matching point pair set, the distance ratio calculation and filtering process is initiated. For each candidate matching point pair, the ratio of its nearest neighbor distance to its second nearest neighbor distance is calculated, and a distance ratio threshold is determined based on preset rules. Subsequently, the initial candidate matching point pair set is filtered according to this threshold, eliminating low-confidence matching point pairs with insufficient similarity discrimination, resulting in a preliminary filtered matching point pair set. Next, a random sampling consensus algorithm is used to perform binocular geometric consistency verification on the preliminary filtered matching point pair set, estimating the binocular spatial geometric constraint relationship and eliminating mismatched point pairs that do not meet the constraints, resulting in a valid matching point pair set. Finally, the total number of valid matching point pairs is counted and compared with the calibration start threshold to generate a judgment conclusion on whether the calibration start condition is met. The final matching results are then obtained by summarizing these results. This hierarchical and progressive filtering mechanism and threshold judgment improve the overall accuracy of matching point pairs and the rationality of the calibration start judgment, avoiding parameter solution deviations caused by mismatches and insufficient data.

[0098] For example, there are two implementation methods for cross-camera feature matching, filtering, and threshold determination. The first method is a unidirectional serial progressive filtering. After the matching and filtering process is officially started, the feature point set of the left-side fisheye image is used as a benchmark. The feature descriptors of the feature points are extracted sequentially. In the right-side feature point set, the nearest and second nearest neighbors with the highest similarity are retrieved. The distance ratio is calculated and compared with the threshold. Matching pairs that meet the requirements are retained. After the retrieval and initial screening of all points are completed, a preliminary set of matched point pairs is obtained. Then, a random sampling consistency algorithm is used to perform a global binocular geometric consistency check on the preliminary set. The geometric deviation of the matching pairs is checked point by point. Mismatched point pairs that exceed the deviation threshold are removed, resulting in a set of valid matching point pairs. Finally, the total number of valid matching point pairs is counted and compared with the calibration start threshold to generate the matching result. This method uses a unidirectional benchmark serial retrieval and progressive hierarchical filtering. The calculation logic is simple and stable, and the intermediate results are highly controllable. It is suitable for conventional calibration scenarios with a moderate number of feature points and clear scene textures.

[0099] The second method is bidirectional parallel block-based constraint filtering. Using the feature point sets of two fisheye images as a benchmark, bidirectional nearest neighbor similarity retrieval is initiated simultaneously, and candidate point retrieval and distance ratio calculation are performed in parallel for both paths. Bidirectional consistency verification filters out corresponding matching pairs, generating an initial filtering set. Subsequently, the overlapping field of view is divided into multiple uniform sub-blocks. Local geometric constraint verification is performed on matching point pairs within each sub-block, eliminating discrete mismatched pairs with excessive offsets. Global random sampling consistency geometric verification is then performed to further eliminate globally abnormal matching pairs, resulting in a set of valid matching point pairs. Finally, the total number of valid matching point pairs and their regional distribution indicators are statistically analyzed and compared with the calibration start threshold and distribution requirements to generate the final matching result. This method employs a parallel filtering logic combining bidirectional mutual verification, block-based local constraints, and global verification. Bidirectional verification reduces the probability of mismatches, and block constraints further filter discrete error points, significantly improving matching accuracy and stability. It is suitable for high-precision calibration scenarios with complex textures and numerous interference features.

[0100] Furthermore, the calibration process of the dual fisheye panoramic camera is completed in seven stages: image acquisition, image preprocessing, feature extraction, feature matching, camera parameter calculation, parameter optimization, and generation of calibration parameters. First, the dual fisheye camera device simultaneously acquires two fisheye images with overlapping fields of view. After the images enter the preprocessing module for grayscale processing, noise reduction, contrast enhancement, and brightness equalization, they enter the feature extraction module to automatically extract natural feature points in the overlapping fields of view and generate corresponding feature descriptors. Then, they enter the feature matching module to obtain effective matching point pairs through nearest neighbor matching and hierarchical screening. In a preferred embodiment, when the number of effective matching points is greater than 50, automatic calibration calculation is initiated. When the number of effective matching points is insufficient, the system automatically expands the search area or accumulates the matching results in multiple consecutive frames of images to improve calibration stability. After obtaining stable matching points, the camera parameter calculation module calculates the dual fisheye camera parameters based on the matching point pairs. The intrinsic parameters include focal length, principal point coordinates, and distortion parameters, while the extrinsic parameters include the rotation matrix and translation vector. The system first establishes binocular geometric constraints based on the matching points, constructing a total reprojection error model consisting of the sum of squared deviations between the actual matching point positions and the reprojection positions calculated using the current camera parameters. Then, in the parameter optimization stage, a Levenberg-Marquardt nonlinear optimization algorithm is used to jointly optimize the intrinsic, extrinsic, and distortion parameters, minimizing the overall reprojection error and ultimately generating the optimal calibration parameters. Unlike traditional methods that rely on calibration boards, this solution utilizes natural features in the actual scene to solve for parameters. It eliminates the need for manual placement of calibration boards and manual acquisition of a large number of calibration images, thereby shortening the equipment installation and debugging time. After calibration, the final calculated intrinsic, extrinsic, and distortion parameters are written to the storage module and made available for use by the panoramic stitching module. During equipment operation, the system continuously performs parameter drift detection, specifically by periodically re-extracting natural feature points from the overlapping area of ​​the dual fisheye images and calculating the reprojection error, average matching distance, and stitching seam continuity index of the current matching point set.

[0101] Fourth embodiment This embodiment provides an exemplary scheme for dynamically determining camera parameter calculation strategies based on matching results. In this example, for matching results that do not meet the calibration start threshold, overlapping field of view expansion or multi-frame matching result accumulation and supplementation are first performed to obtain a set of qualified matching point pairs that meet the calibration start conditions. Then, the pixel coordinate correspondence and confidence weight of the matching point pairs are extracted to generate a matching point coordinate mapping dataset with confidence weights. Subsequently, according to the binocular imaging parameter calculation rules, the spatial geometric constraint relationship of the dual lenses is constructed and the camera parameter intervals participating in the calculation are defined. Finally, the parameter calculation strategy with minimizing the weighted projection error as the convergence objective is determined, thereby providing an accurate execution basis for the subsequent construction of the reprojection error model and parameter optimization solution. Step S30 includes steps C11~C14: Step C11: If the matching result does not meet the calibration start threshold, then expand the overlapping field of view search area or accumulate the matching results to obtain a set of qualified matching point pairs that meet the calibration start conditions.

[0102] Step C12: Based on the set of qualified matching point pairs, extract the pixel coordinate correspondence and confidence weight of each matching point pair to obtain a matching point coordinate mapping dataset including the confidence weight.

[0103] Step C13: Based on the binocular imaging parameter calculation rules and the matching point coordinate mapping dataset, construct the spatial geometric constraint relationship of the dual lenses and define the camera parameter ranges participating in the calculation.

[0104] Step C14: Based on the dual-lens spatial geometric constraints and the camera parameter range, determine the parameter calculation strategy with the goal of minimizing the weighted projection error.

[0105] The qualified matching point pair set is a collection of valid matching point pairs that, after quantity supplementation and quality verification, meet the calibration start-up data scale requirements. It serves as the core data foundation for subsequent parameter calculations. The matching point coordinate mapping dataset is a structured dataset that integrates the dual-path pixel coordinate correspondence and corresponding confidence weights of all matching point pairs. It is the direct input object for constructing spatial geometric constraints. The dual-lens spatial geometric constraint relationship is the spatial pose constraint condition between the two lenses, derived from the matching point coordinate mapping relationship based on the principle of binocular vision imaging. It is the core theoretical basis for solving camera parameters.

[0106] The camera parameter range is the range of camera parameter categories involved in this calculation and the initial value boundary of each parameter; it serves as the value constraint for parameter iterative optimization. The parameter calculation strategy is the overall execution plan for this calibration task, defining the parameter solution range, optimization objective, weighting rules, and iterative constraints. It serves as the guiding rule for subsequently building the error model and performing parameter optimization. For example, the number of qualified matching point pairs is not less than the calibration initiation threshold and is spatially uniform; the matching point coordinate mapping dataset contains three core fields: left-side pixel coordinates, right-side pixel coordinates, and corresponding confidence weights; and the camera parameter range can cover three types of parameters: camera intrinsics, distortion parameters, and relative extrinsic parameters between the two lenses.

[0107] In this example, when supplementing matching results that do not meet the calibration threshold to obtain a set of qualified matching point pairs, this can be done by expanding the search area of ​​the overlapping field of view. That is, the search boundary of the overlapping field of view is expanded outwards, incorporating feature points from the original edge regions into the matching calculation range. Feature matching and filtering are then re-executed to supplement the number of valid matching point pairs until the calibration threshold is met. Alternatively, the method can be used to accumulate matching results across multiple frames. Valid matching point pairs are collected frame by frame from consecutive fisheye images, and the stable matching results from multiple frames are accumulated and summarized until the total number of point pairs meets the calibration threshold, thus completing the construction of the set of qualified matching point pairs.

[0108] After constructing the set of qualified matching point pairs, the matching point coordinate mapping dataset generation process is initiated. For each matching point pair within the set, the pixel coordinate correspondence between the two fisheye images is extracted. Simultaneously, a corresponding confidence weight is assigned based on the distortion degree and matching confidence of the matching point's region, resulting in a weighted matching point coordinate mapping dataset. Next, based on the binocular imaging parameter calculation rules and the matching point coordinate mapping dataset, the spatial pose correspondence between the two lenses is derived, constructing a dual-lens spatial geometric constraint relationship. Furthermore, based on the calibrated scene type and current parameter state, the camera parameter categories participating in this calculation and the initial value ranges for each parameter are defined. Finally, based on the dual-lens spatial geometric constraint relationship and camera parameter ranges, a convergence objective centered on minimizing the weighted projection error is set, along with corresponding iteration rules and constraints, forming a complete parameter calculation strategy. This improves the adaptability and convergence stability of parameter solutions through a pre-processing data supplementation mechanism and customized parameter strategy settings, avoiding solution deviations caused by insufficient data or unreasonable parameter ranges.

[0109] For example, there are two methods for determining the parameter calculation strategy. The first is a preset strategy matching method, which pre-stores multiple sets of standard parameter calculation strategies adapted to different matching scales and calibration scenarios. Each strategy corresponds to the matching conditions of the number of matching points, distribution uniformity, and calibration type. The number of point pairs, distribution uniformity, average confidence, and other indicators of the current qualified matching point pair set are compared with the matching conditions of each strategy one by one, and the most suitable standard calculation strategy is directly matched and selected as the parameter calculation strategy for this calibration. This method relies on direct matching based on preset strategies, has simple logic, fast response speed, and the strategy has been verified to have high stability in a large number of scenarios, making it suitable for regular periodic calibration and rapid calibration scenarios.

[0110] The second method is dynamic customized generative determination. It extracts core features from the current set of qualified matching point pairs, such as the number of point pairs, spatial distribution density, and region confidence weights. Combined with the binocular imaging parameter calculation rules, it dynamically adjusts the category range, initial value interval, weight allocation rules, and convergence accuracy requirements of the parameters to be solved, generating a customized parameter calculation strategy that is fully adapted to the current matching data. Simultaneously, it verifies the feasibility of the generated strategy, confirming that the parameter range and convergence rules conform to the imaging geometric constraints. After verification, the output is the parameter calculation strategy for this calibration. This method allows for flexible strategy customization based on actual matching data, resulting in higher adaptation accuracy. It can fully exploit the effective information in the matching data, improve the accuracy of parameter solving, and is suitable for high-precision calibration and calibration in complex scenarios.

[0111] Furthermore, the dual fisheye panoramic camera continuously performs parameter drift detection during device operation. The system periodically re-extracts natural feature points from the overlapping field of view of the two fisheye images and completes the matching. It calculates the reprojection error of the current set of matched points, the average offset of the matched points, and the continuity index of the stitching seam. The average reprojection error is defined as the arithmetic mean of the reprojection errors of all matched points, i.e., Erroravg=(1 / N)ΣEi, where Ei represents the reprojection error of the i-th matched point, and N represents the number of valid matched points. When any of the following conditions are met: the average reprojection error exceeds a preset threshold, the average offset of the matched points exceeds a preset threshold, continuous misalignment occurs in the stitching seam area, or the error continues to increase over multiple consecutive frames, it is determined that the camera parameters have drifted. The error threshold can be set from 1 pixel to 5 pixels depending on the device resolution. Upon detecting parameter drift, the system automatically initiates a parameter correction process. This correction process does not involve a complete recalibration; instead, it uses the currently stored camera calibration parameters as initial values ​​and performs incremental optimization calculations only on the rotation matrix R and translation vector T of the extrinsic parameters. The system employs a sliding time window mechanism, statistically analyzing stable matching points in the most recent N frames. The parameter correction amounts for rotation increment ΔR and translation increment ΔT are calculated using a least-squares optimization algorithm. The relative extrinsic parameters of the two lenses are then updated according to the rules R'=R+ΔR and T'=T+ΔT. The updated parameters are immediately used in subsequent panoramic stitching processing. To avoid erroneous corrections due to abnormal scenarios, the system incorporates a parameter reliability evaluation mechanism. Only when consistent correction results are obtained across multiple consecutive time windows are the updated camera calibration parameters written to the storage module; otherwise, the original parameter configuration is maintained. Through this approach, this solution can automatically detect parameter drift and correct it in real time when the equipment is subjected to temperature changes, mechanical vibrations, mounting bracket deformation, or minor changes in lens attitude due to long-term operation. This ensures the long-term accuracy and continuity of the panoramic stitched images without the need for manual recalibration.

[0112] Fifth Embodiment This embodiment provides an exemplary scheme for constructing a weighted projection error model based on a parameter operation strategy. In this example, the range of parameters to be solved, the weight allocation rules, and the optimization convergence objective in the parameter operation strategy are first analyzed. A basic modeling dataset is generated by combining the actual observed pixel coordinates of the matching points. Then, a point-by-point reprojection calculation logic is built based on the fisheye imaging geometry principle and the spatial geometric constraints of dual lenses. Subsequently, a corresponding projection confidence weight is assigned to each set of matching points, and an error calculation model is constructed with the weighted sum of reprojection deviations as the optimization objective. Finally, the parameter constraints and operational boundaries of the model are verified, thereby outputting a reprojection error model that meets the requirements of the operation strategy. Step S40 includes steps D11~D14: Step D11: Analyze the range of parameters to be solved, weight allocation rules, and optimization convergence objectives in the parameter calculation strategy, and generate the basic dataset for modeling by combining the actual observed pixel coordinates of all matching point pairs.

[0113] Step D12: Based on the fisheye imaging geometry principle, the spatial geometric constraints of dual lenses, and each pair of matching points in the modeling dataset, construct the reprojection calculation logic for pixel coordinates and reprojection coordinates.

[0114] Step D13: Match the corresponding projection confidence weight for the reprojection calculation logic of each pair of matching points, and construct an initial calculation model with the weighted sum of the reprojection deviations of all matching points as the optimization objective.

[0115] Step D14: Verify the parameter constraints and operation boundaries of the initial calculation model. After confirming that it meets the requirements of the parameter operation strategy, output the reprojection error model.

[0116] The modeling foundation dataset is a structured dataset integrating the core configuration of parameter calculation strategies and the actual observed coordinates of all matching points. It serves as the fundamental input data source for constructing the reprojection error model. The reprojection calculation logic describes the complete computational chain from the actual pixel coordinates of a single set of matching points to the reprojected coordinates via camera parameter transformation; it is the core computational unit of the error model. Projection confidence weights are computational weights allocated based on the quality of the matching points and the degree of regional distortion, used to differentiate the contribution of different matching points to the overall error. The initial computational model is a mathematical computational model with the weighted sum of reprojection deviations of all matching points as the optimization objective; it is the core solution basis for subsequent parameter iterative optimization. Parameter constraints are rules that limit the range and geometric rationality of the parameters to be solved, used to ensure the stability and effectiveness of the iterative solution process. The computational boundary is the legal boundary range of parameter values ​​and error values ​​during model calculation, used to avoid invalid calculations and iterative divergence. For example, the basic dataset for modeling includes a list of parameters to be solved, weight allocation rules, convergence target value and dual-path pixel coordinates of all matching points. The reprojection calculation logic covers four calculation steps: distortion correction, intrinsic parameter transformation, extrinsic parameter spatial transformation and secondary projection. The projection confidence weight decreases step by step from the central stable region to the edge distortion region.

[0117] In this example, when parsing the parameter calculation strategy and combining it with the coordinates of matching points to generate the basic modeling dataset, a full synchronous parsing approach can be used. This involves reading all configuration items of the parameter calculation strategy at once, extracting the range of parameters to be solved, weight allocation rules, and optimization convergence objectives, and simultaneously importing the actual observed pixel coordinates of all matching point pairs in batches. After data integration and verification, the basic modeling dataset is output uniformly. Alternatively, a streaming incremental parsing approach can be used, where the configuration items of the parameter calculation strategy are read and parsed simultaneously to extract the core rules. The pixel coordinates of matching point pairs are loaded group by group and data verification is performed, and the basic modeling dataset is generated group by group, thus completing the collection of basic modeling data.

[0118] After constructing the basic modeling dataset, the reprojection calculation logic was built. For each pair of matching points in the dataset, based on the fisheye imaging geometry and dual-lens spatial geometric constraints, a complete calculation chain was constructed, including pixel coordinate distortion correction, camera intrinsic parameter transformation, dual-lens relative extrinsic parameter spatial transformation, and projection imaging from the other lens, resulting in point-by-point reprojection calculation logic. Subsequently, a corresponding projection confidence weight was assigned to the reprojection calculation logic of each pair of matching points. The reprojection deviations of all matching points were multiplied by their corresponding weights and summed. The weighted total deviation was used as the optimization objective to integrate and form the initial calculation model. Finally, the model's parameter constraints and operational boundaries were verified to ensure that all constraint rules and boundary ranges met the requirements of the parameter operation strategy. After verification, the final reprojection error model was output. This layered chain construction and differentiated weight configuration improved the calculation accuracy and optimization guidance of the error model, avoiding interference from edge distortion points caused by uniform weights.

[0119] For example, there are two ways to construct the reprojection error model. The first is to construct a globally uniform weight model. Based on the parameters to be solved and the fisheye imaging projection logic defined by the parameter calculation strategy, a reprojection calculation link is established, which transforms pixel coordinates through camera intrinsic parameters, distortion parameters, and extrinsic parameters. The same calculation weight is assigned to all matching point pairs. The arithmetic mean of the reprojection deviation of all matching points is used as the optimization objective to construct a globally uniform weight reprojection error model. After completing parameter constraints and calculation boundary verification, the model is output. This method has a simple model structure, clear calculation logic, and strong iterative solution stability. It is suitable for conventional calibration scenarios where matching points are evenly distributed and scene imaging quality is consistent.

[0120] The second approach involves constructing a region-specific adaptive weighted model. Combining fisheye distortion distribution with the confidence level of matching points, the model divides matching point pairs into three levels: high-confidence, medium-confidence, and low-confidence regions. Calculation weights are assigned to matching points at different levels with progressively decreasing weights. A distortion-sensitive term is introduced, adding distortion correction weight factors for matching points in distorted edge regions, thus correcting the reprojection deviation calculation logic in these areas. Finally, the total weighted projection deviation of all matching points is used as the optimization objective to construct a region-specific adaptive weighted reprojection error model, which is then output after multi-dimensional constraint verification. This method fully considers the distortion characteristics of fisheye imaging and the quality differences of matching points, guiding the optimization direction through differentiated weights. It effectively improves the overall accuracy of parameter solving and is suitable for high-precision calibration and complex imaging scenarios.

[0121] Sixth Embodiment This embodiment provides an exemplary scheme for solving camera calibration parameters based on a reprojection error model and joint nonlinear optimization. In this example, the reprojection error model first extracts the parameter optimization convergence target, the set of parameters to be optimized, and the matching confidence weights. The initial iteration value and step size constraint range of the nonlinear optimization algorithm are set, and a parameter iteration solution configuration is generated. Then, based on the solution configuration, the nonlinear optimization algorithm performs joint iterative calculations on the camera intrinsic parameters, distortion parameters, and relative extrinsic parameters of the two lenses to obtain a globally weighted projection error value. Subsequently, the error value output in each iteration is compared with a preset convergence threshold. When the error value is below the threshold for multiple consecutive iterations and the fluctuation amplitude meets the standard, the optimal parameter set is output. Finally, an imaging geometric consistency check is performed on the optimal parameter set to obtain a complete set of standard format camera calibration parameters. Step S50 includes steps E11~E14: Step E11: By setting the initial iteration value and step size constraint range of the nonlinear optimization algorithm through the parameter optimization convergence objective, the set of parameters to be optimized, and the matching confidence weights in the reprojection error model, a parameter iteration solution configuration is generated.

[0122] Step E12: Based on the parameter iterative solution configuration and the nonlinear optimization algorithm, perform joint iterative calculations on the camera intrinsic parameters, distortion parameters, and relative extrinsic parameters of the dual lenses to obtain the global weighted projection error value.

[0123] Step E13: Compare the global weighted projection error value output in each iteration with the preset convergence threshold. When the global weighted projection error value is lower than the preset convergence threshold for a preset number of consecutive iterations and the fluctuation amplitude is less than the set precision, output the current optimal parameter set.

[0124] Step E14: Perform imaging geometry consistency verification on the optimal parameter set to obtain the camera calibration parameters in a complete standard format.

[0125] The parameter iteration configuration is an optimized solution configuration set that integrates initial iteration values, step size constraints, maximum number of iterations, and convergence criteria. It serves as the execution basis for nonlinear optimization algorithms to solve parameters. The global weighted projection error is the sum of the reprojection deviations of all matched point pairs multiplied by their corresponding matching confidence weights. It is a core quantitative indicator for evaluating the quality of current parameter values. The optimal parameter set is the combination of camera parameter values ​​that minimizes the overall reprojection error after the iteration process satisfies the convergence condition. It is the pre-output result of the final calibration parameters. Imaging geometric consistency verification is a verification process based on the binocular vision imaging principle to verify the physical rationality of parameter values, used to exclude mathematically optimal but geometrically invalid parameter results. The camera calibration parameters are a standardized set of complete camera imaging and pose parameters obtained after solving and verification. They are the core parameters called for subsequent panoramic stitching. For example, the parameter iteration solution configuration includes the initial values ​​of camera intrinsic parameters, the initial values ​​of distortion parameters, the initial values ​​of extrinsic parameters, the upper and lower limits of the iteration step size, the maximum number of iterations and the convergence threshold. The global weighted projection error value continues to decrease with parameter optimization until convergence. The camera calibration parameters include three core contents: camera intrinsic parameters, distortion parameters and dual-lens relative extrinsic parameters.

[0126] In this example, when setting the initial values ​​and step size constraints for the nonlinear optimization algorithm to generate the parameter iteration solution configuration, it can be done by initializing the parameters according to the factory baseline parameters. That is, the factory nominal parameters of the fisheye lens are retrieved as the initial values ​​for iteration, the step size constraint boundary is set in combination with the reasonable range of the parameters' physical properties, and the parameter iteration solution configuration is generated after matching the corresponding convergence threshold. Alternatively, it can be done by dynamically initializing the parameters using historical calibration parameters. The previous version of valid calibration parameters stored in the device is used as the initial values ​​for iteration, the step size constraint range is dynamically narrowed according to the parameter drift amplitude, and the parameter iteration solution configuration is generated after matching the convergence threshold of the corresponding accuracy. This completes the generation of the parameter iteration solution configuration.

[0127] After generating the parameter iterative solution configuration, a nonlinear optimization joint iterative process is initiated. Based on the solution configuration and the nonlinear optimization algorithm, synchronous joint iterative calculations are performed on the camera intrinsic parameters, distortion parameters, and relative extrinsic parameters of the dual lenses. In each round, after updating all parameter values ​​to be optimized, the current global weighted projection error value is calculated synchronously. Subsequently, the error value output in each round is compared with a preset convergence threshold, and the error fluctuation amplitude over multiple rounds is monitored. When the error value is below the threshold for a preset number of consecutive rounds and the fluctuation amplitude is less than the set precision, the iteration terminates and the current optimal parameter set is output. Finally, an imaging geometry consistency check is performed on the optimal parameter set. After verifying that the parameter values ​​conform to the binocular spatial imaging rules, the entire set of camera calibration parameters is output in a standard format. This joint iterative optimization and hierarchical convergence check improve the accuracy and reliability of the parameter solution, avoiding iterative divergence or geometrically invalid parameter output.

[0128] For example, there are two methods for optimizing camera calibration parameters. The first is a fixed-step global iterative solution. Using the initial value set in the parameter iterative solution configuration as the starting point, a nonlinear optimization algorithm is employed to perform global parameter iteration with a fixed step size. In each round, the values ​​of the camera's intrinsic parameters, distortion parameters, and relative extrinsic parameters between the two lenses are updated synchronously, and the current global weighted projection error value is calculated. When the error value is below the convergence threshold for several consecutive rounds and the fluctuation amplitude is less than the set accuracy, the iteration terminates and the optimal parameter set is output. After verification, the camera calibration parameters are generated. This method offers a stable and controllable solution process, simple iteration rules, and is less prone to iteration divergence problems, making it suitable for calibration scenarios with conventional accuracy requirements.

[0129] The second approach is adaptive step-size hierarchical iterative solution. This method divides the optimization process into two stages: coarse convergence and fine convergence. In the coarse convergence stage, a larger iteration step size is used to quickly narrow the parameter value range, causing the global weighted projection error to rapidly decrease to near the optimal range. In the fine convergence stage, the iteration step size is automatically reduced, performing fine adjustments to all parameters while gradually tightening the convergence threshold. During the iteration process, the error reduction trend is monitored in real time. If the error continues to rise, the parameters are automatically reverted and the step size is reduced to ensure stable convergence. Once the convergence condition is met, the optimal parameter set is output, and after verification, camera calibration parameters are generated. This method balances solution speed and accuracy through hierarchical iteration, and the adaptive step size effectively avoids iterative oscillations, improving solution efficiency and final accuracy. It is suitable for high-precision calibration and multi-parameter joint optimization scenarios.

[0130] Further, please refer to Figure 4 , Figure 4This is a schematic diagram of the dual fisheye panoramic camera system structure of this application. This embodiment is applicable to a panoramic camera system with two fisheye lenses. The system consists of hardware devices and software functional modules. The hardware includes a dual fisheye camera device, image sensor 1, image sensor 2, an image processing unit, and a storage module. The dual fisheye camera device is equipped with two fisheye lenses, fisheye lens 1 and fisheye lens 2. Each lens is connected to one image sensor. The two lenses are arranged on the same device in opposite or back-to-back directions. The field of view of a single lens is approximately 180°, and there is a certain overlap between the fields of view of the two lenses. The image processing unit can be implemented using a central processing unit, graphics processing unit, or neural network processor. It is used to carry and execute all image processing algorithms and parameter calculation programs, including image acquisition, feature extraction, parameter calculation, and stitching processing. The storage module is used to persistently store camera calibration parameters and correction parameters generated during device operation. The system's software modules all run on the image processing unit, specifically including an image acquisition module, an image preprocessing module, a feature extraction module, a feature matching module, a camera parameter calculation module, a panoramic stitching module, a parameter drift detection module, an automatic parameter correction module, and a parameter storage module. In the main workflow, the raw images acquired by the dual fisheye cameras are output from the image acquisition module and then sequentially flow through the image preprocessing module, feature extraction module, feature matching module, and camera parameter calculation module to solve for calibration parameters. Finally, they are sent to the panoramic stitching module to complete the panoramic image stitching process. Simultaneously, two image data streams are input to the parameter drift detection module. After the automatic parameter correction module determines parameter drift and performs incremental correction of extrinsic parameters, the updated parameters that pass verification are written to the parameter storage module for finalization. The entire system automatically extracts natural feature points from the overlapping field of view of the dual fisheye images to calculate camera calibration parameters and continuously monitors the feature matching status and stitching accuracy changes during equipment operation. This enables automated online correction of camera parameters, ensuring the long-term alignment accuracy and image continuity of the panoramic stitched images.

[0131] Seventh Embodiment This embodiment provides an exemplary scheme for online detection and judgment of camera parameter drift. In this example, in response to parameter correction instructions, real-world natural feature points in the overlapping field of view of two fisheye images are extracted. The reprojection correspondence of feature points is calculated according to the calibration parameter correction rules to obtain a set of projection matching point pairs. Then, combined with the currently effective camera calibration parameters, reprojection operations are performed point by point to generate three types of indicators: average reprojection error, average offset of matching points, and continuity of stitching seams. Subsequently, each detection indicator is compared with its corresponding threshold, and the error value change trajectory of multiple consecutive frames is traced. Finally, the parameter drift state is determined based on whether the indicators exceed limits or the error continues to increase, and the parameter drift judgment result is output. Please refer to... Figure 5 , Figure 5This is a flowchart illustrating the seventh embodiment of the calibration method for a dual fisheye panoramic camera according to this application. Following step S50, steps F11-F14 are also included: Step F11: In response to the parameter correction command, extract the real-world natural feature points of the overlapping field of view of the two fisheye images, and calculate the reprojection correspondence of the feature points according to the calibration parameter correction rules to obtain a set of projection matching point pairs.

[0132] Step F12: Based on the set of projection matching points, perform reprojection calculations point by point in conjunction with the currently effective camera calibration parameters to generate the average reprojection error, the average offset of matching points, and the continuity of the stitching seam.

[0133] Step F13: Compare the average reprojection error, the average offset of the matching points, and the continuity of the splicing seam with the corresponding index thresholds one by one, and trace the error value change trajectory of multiple consecutive frames to obtain the index detection results.

[0134] Step F14: When any of the detected indicators in the indicator detection results exceeds the indicator threshold, or when a continuous increasing trend of error is detected in multiple consecutive frames, it is determined that the camera parameters have drifted, and the parameter drift judgment result is output.

[0135] The parameter correction command is the control command that triggers the parameter drift detection and correction process, used to initiate periodic parameter verification or abnormally triggered parameter verification. The projection matching point pair set is the feature point pairing result based on the current calibration parameters, which completes reprojection mapping and establishes spatial correspondence; it is the fundamental data for calculating drift detection indicators. The average reprojection error is the arithmetic mean of the reprojection deviations of all matching points, a core quantitative indicator for measuring the overall deviation of camera parameters. The average offset of matching points is the average coordinate offset between the actual position and the reprojected position of all matching points, a direct indicator of the degree of lens pose offset. The stitching seam continuity is the smoothness of the pixel transition in the seam area after panoramic stitching, a direct indicator reflecting the stitching alignment effect. The indicator detection result is a comprehensive detection conclusion obtained after completing all indicator comparisons and trend analysis, including the compliance status and error change trend of each indicator. The parameter drift judgment result is the final output parameter status judgment conclusion, divided into two categories: parameter drift occurred and parameter no drift occurred. For example, parameter correction instructions can be triggered by timing cycles, splicing quality anomalies, or sudden changes in equipment temperature. The threshold for the average reprojection error is typically set to 1 to 5 pixels.

[0136] In this example, when extracting real-world natural feature points and calculating reprojection correspondences to obtain a set of projection matching point pairs, the process can be performed in real-time extraction and matching for each frame. Specifically, for the two fisheye images of the current frame, real-world natural feature points in the overlapping field of view are directly extracted. Reprojection mapping and feature matching are then performed using the currently effective calibration parameters to obtain a single-frame set of projection matching point pairs. Alternatively, a sliding window multi-frame accumulation method can be used. Starting from the current frame, a preset number of frames are traced back, feature points are extracted frame by frame, and corresponding matching relationships are calculated. Matching point pairs that are consistently present across multiple frames are selected, and the accumulated set of projection matching point pairs is obtained, thus completing the acquisition of projection matching point pair data.

[0137] After constructing the set of projection matching point pairs, the reprojection operation and index generation process is initiated. Combining the currently effective camera calibration parameters, reprojection operations are performed point-by-point on each matching point pair within the set. The reprojection deviation and coordinate offset of each point are calculated, and the average reprojection error and average offset of the matching points are statistically obtained. Simultaneously, the continuity of the stitching seam is calculated based on the distribution of matching points in the stitching seam area. Then, the three indices are compared one by one with preset corresponding threshold values. Historical error data from multiple consecutive frames is retrieved to trace the trajectory of error value changes, identify the trend of continuous error growth, and summarize the complete index detection results. Finally, a drift judgment is performed based on the index detection results. If any detected index exceeds the corresponding threshold, or if a continuous upward trend in error is detected across multiple consecutive frames, camera parameter drift is determined; otherwise, the parameter status is considered normal, and the final parameter drift judgment result is output. This multi-dimensional index joint judgment combined with trend tracing improves the accuracy of parameter drift detection and avoids misjudgments caused by single-frame anomalies or local interference.

[0138] For example, there are two implementation methods for camera parameter drift detection and determination. The first is single-frame threshold-triggered detection. After the detection process starts, three indicators are calculated based on the set of projection matching point pairs in the current frame: average reprojection error, average offset of matching points, and continuity of stitching seams. Each of the three indicators is compared with a preset threshold. If any indicator exceeds the threshold, the parameter is directly determined to have drifted. If all indicators are within the threshold range, the parameter is determined to be normal, and the parameter drift determination result is output. This method has a simple and direct detection logic, can complete the determination in a single frame, has a fast response speed, and is suitable for regular periodic parameter verification scenarios.

[0139] The second method is multi-frame trend-linked detection. First, based on multiple frames within a sliding time window, three detection indicators are calculated frame-by-frame, generating a continuous sequence of indicators across multiple frames. This sequence compares the individual frame indicators with thresholds to determine if they exceed limits, and analyzes the trend of the indicator sequence to identify drift characteristics indicating a continuously increasing error. When an individual frame indicator exceeds its limit, or when the error shows a stable upward trend across multiple frames, camera parameter drift is determined. If all frame indicators meet the standards and show no continuous upward trend, the parameters are considered normal, and the parameter drift judgment result is output. This method supplements single-frame threshold determination with multi-frame trend analysis, effectively identifying gradual, minute parameter drifts while shielding against interference from abnormal single-frame scenes. This significantly improves detection sensitivity and anti-interference capabilities, making it suitable for long-term online monitoring scenarios requiring high precision.

[0140] Eighth embodiment This embodiment provides an exemplary scheme for incremental online correction of relative extrinsic parameters of dual lenses. In this example, an incremental parameter correction strategy is first initiated based on the parameter drift judgment result. The currently effective camera calibration parameters are used as the initial values ​​for iteration, keeping the camera intrinsic and distortion parameters constant. A sliding time window mechanism is used to accumulate matching point pairs from multiple consecutive frames to generate a dedicated matching point dataset for correction. Then, using the relative extrinsic parameters of the dual lenses as the sole optimization variable, a least-squares optimization algorithm is employed to solve for the rotation and translation increments of the extrinsic parameters, obtaining the incremental correction amount. Finally, the relative extrinsic parameters of the dual lenses are updated based on the correction amount, and the updated target camera calibration parameters are loaded into the panoramic stitching processing module. This allows for rapid parameter drift correction without restarting the full calibration, ensuring the alignment accuracy of subsequent image stitching. Please refer to... Figure 6 , Figure 6 This is a flowchart illustrating the eighth embodiment of the calibration method for the dual fisheye panoramic camera of this application. Following step F14, steps G11-G13 are also included: Step G11: Based on the parameter drift judgment result, an incremental parameter correction strategy is initiated. The camera calibration parameters are used as the initial values ​​for iteration, and the distortion parameters involved in the camera are kept unchanged. The matching point pairs of multiple consecutive frames are accumulated through a sliding time window mechanism to generate a correction-specific matching point dataset.

[0141] Step G12: Based on the modified dedicated matching point dataset, using the relative extrinsic parameters of the two lenses as optimization variables, calculate the rotation increment and translation increment of the extrinsic parameters according to the least squares optimization algorithm to obtain the extrinsic parameter increment correction amount.

[0142] Step G13: Update the relative extrinsic parameters of the dual lenses according to the extrinsic parameter increment correction amount, and load the updated target camera calibration parameters into the panoramic stitching processing module for subsequent image stitching calculations.

[0143] The incremental parameter correction strategy is a correction execution strategy that targets minute parameter drifts, optimizing only the incremental extrinsic parameters rather than performing a full recalibration. It is the core execution principle for online fast correction. The dedicated matching point dataset for correction is a set of valid matching point pairs that are stable across multiple frames, accumulated through a sliding time window. It is the core data source for calculating incremental extrinsic parameters. The incremental extrinsic parameter correction amount is used to correct the adjustment of the current dual-lens relative extrinsic parameters. It consists of two parts: rotation increment and translation increment, and is the core calculation result for implementing incremental parameter updates. The rotation increment is the adjustment amount of the rotation matrix in the dual-lens relative extrinsic parameters, used to correct the angular offset of the lens attitude. The translation increment is the adjustment amount of the translation vector in the dual-lens relative extrinsic parameters, used to correct the offset of the lens spatial position. The target camera calibration parameters are the complete set of camera calibration parameters after the extrinsic parameter update, and are the effective parameters directly used in the stitching module. For example, the incremental parameter correction strategy only adjusts the rotation matrix and translation vector, without changing intrinsic parameters such as focal length, principal point, and distortion coefficient; the correction-specific matching point dataset contains the dual-path pixel coordinates and confidence weights of all stable matching point pairs within the sliding window.

[0144] In this example, when accumulating matching point pairs using a sliding time window mechanism to generate a correction-specific matching point dataset, a fixed-length window accumulation method can be used. This involves pre-setting a fixed number of window frames, traversing back the corresponding number of frames from the current frame as the end of the window, filtering stable matching point pairs frame by frame, summarizing all valid matching points, removing duplicates, and merging them to generate the correction-specific matching point dataset. Alternatively, an adaptive-length window accumulation method can be used. This dynamically adjusts the window length based on the average confidence of the current matching points. When the confidence of a matching point is high, the window is shortened to improve the correction response speed; when the confidence is low, the window is extended to improve data stability. This dynamic accumulation yields the correction-specific matching point dataset, thus completing the preparation of the correction data.

[0145] After constructing the dedicated matching point dataset, the incremental optimization process for extrinsic parameters is initiated. Using the relative extrinsic parameters of the two lenses as the sole optimization variable, an incremental optimization objective is constructed based on the least squares optimization algorithm. The rotation and translation increments that minimize the overall reprojection error are iteratively solved, and these are summarized to form the incremental extrinsic parameter correction. Subsequently, the relative extrinsic parameters of the two lenses are incrementally updated based on the incremental extrinsic parameter correction, while keeping the camera's intrinsic and distortion parameters constant. This results in the updated target camera calibration parameters. Finally, the target camera calibration parameters are loaded into the panoramic stitching processing module in real time and directly used for subsequent panoramic image stitching operations. This approach achieves rapid online correction by optimizing only the incremental extrinsic parameters, avoiding the computational overhead of full recalibration and improving the real-time performance and operational efficiency of parameter correction.

[0146] For example, there are two ways to implement incremental extrinsic parameter correction. The first is a single-window least squares direct solution update. After the correction process officially starts, based on the correction-specific matching point dataset within the current sliding time window, and using the current extrinsic parameters as initial values, the least squares optimization algorithm is used to solve for the rotation increment and translation increment that minimize the overall reprojection error in one go. After obtaining the extrinsic parameter incremental correction amount, the relative extrinsic parameters of the two lenses are directly updated, the target camera calibration parameters are generated, and loaded into the stitching module. This method has simple calculation logic, fast solution speed, and timely correction response, and is suitable for conventional online correction scenarios with small drift amplitude and stable scenes.

[0147] The second method is multi-window weighted iterative smoothing update. It sequentially retrieves matching point datasets from multiple consecutive sliding time windows, assigning time weights decreasing from near to far to the matching data in different time windows. Based on the weighted full matching data, multiple rounds of least squares iterative solutions are performed, gradually converging to obtain a smooth and stable incremental correction amount for extrinsic parameters. Then, the relative extrinsic parameters of the two lenses are updated according to the correction amount, generating the target camera calibration parameters and loading them into the stitching module. This method, through multi-window weighting and iterative smoothing, can effectively shield against interference from single-frame anomalies and scene fluctuations, resulting in more stable and reliable correction results. It avoids stitching image jumps caused by single correction deviations, making it suitable for long-term online operation scenarios with high precision requirements.

[0148] Further, please refer to Figure 7 , Figure 7 This is a flowchart of the automatic parameter correction process for this application. This implementation realizes an automated control process for a dual fisheye panoramic camera during operation, triggering parameter updates and recalculating camera extrinsic parameters by detecting feature matching errors: When the device is in normal operating condition, it continuously performs image acquisition operations, extracts feature content from the overlapping area of ​​the two acquired fisheye images, calculates the feature matching error based on the extracted feature data, and compares this error value with a preset threshold. If the feature matching error does not exceed the threshold, the device continues to use the currently effective stitching parameters for subsequent image stitching operations. If the feature matching error exceeds the threshold, a parameter update process is triggered, recalculating the camera extrinsic parameters corresponding to the dual lenses, adjusting and updating the stitching parameters based on the updated extrinsic parameters, and finally saving the generated new parameters, completing the complete closed-loop process of extrinsic parameter recalculation and parameter iterative update driven by excessive matching errors.

[0149] This application provides a calibration device for a dual fisheye panoramic camera. The calibration device for a dual fisheye panoramic camera includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the calibration method for the dual fisheye panoramic camera in the first embodiment described above.

[0150] The following is for reference. Figure 8 The diagram illustrates a structural schematic of a calibration device suitable for implementing the embodiments of this application for a dual fisheye panoramic camera. The calibration device for the dual fisheye panoramic camera in the embodiments of this application may include, but is not limited to, mobile terminals such as high-precision displacement positioning devices, industrial camera brackets, industrial image acquisition cards, etc., as well as fixed terminals such as circular array calibration plates, three-dimensional calibration targets, etc. Figure 8 The calibration device for the dual fisheye panoramic camera shown is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.

[0151] like Figure 8 As shown, the calibration device for a dual-fisheye panoramic camera may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the dual-fisheye panoramic camera calibration device. The processing unit 1001, the read-only memory 1002, and the RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to I / O interface 1006: input devices 1007 including, for example, touchscreens, touchpads, keyboards, mice, image sensors, microphones, accelerometers, gyroscopes, etc.; output devices 1008 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 1003 including, for example, magnetic tapes, hard disks, etc.; and communication devices 1009. Communication device 1009 allows the calibration equipment of the dual fisheye panoramic camera to communicate wirelessly or wiredly with other devices to exchange data. Although calibration equipment for dual fisheye panoramic cameras with various systems is shown in the figures, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.

[0152] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from read-only memory 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.

[0153] The calibration device for a dual fisheye panoramic camera provided in this application, employing the calibration method for a dual fisheye panoramic camera as described in the above embodiments, can solve the technical problem of poor calibration results. Compared with the prior art, the beneficial effects of the calibration device for a dual fisheye panoramic camera provided in this application are the same as those of the calibration method for a dual fisheye panoramic camera provided in the above embodiments, and other technical features in this calibration device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.

[0154] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.

[0155] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0156] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the calibration method of the dual fisheye panoramic camera in the above embodiments.

[0157] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, radio frequency (RF), etc., or any suitable combination thereof.

[0158] The aforementioned computer-readable storage medium may be included in the calibration equipment of the dual fisheye panoramic camera; or it may exist independently and not assembled into the calibration equipment of the dual fisheye panoramic camera.

[0159] The aforementioned computer-readable storage medium carries one or more programs. When these programs are executed by the calibration device of a dual-fisheye panoramic camera, the calibration device performs the following actions: after obtaining two fisheye images from the dual-fisheye panoramic camera, it extracts real-world natural feature points from the overlapping field of view of the two fisheye images and generates feature descriptors corresponding to these real-world natural feature points; it performs cross-lens feature matching on the feature descriptors using a nearest neighbor matching algorithm, filters valid matching point pairs, and obtains matching results; it determines the parameter operation strategy corresponding to the matching results based on the matching results and the binocular imaging parameter calculation rules; it constructs a reprojection error model based on the parameter operation strategy and the positional correspondence of the matching point pairs; it determines the parameter optimization convergence target through the reprojection error model and solves for the camera calibration parameters using a nonlinear optimization algorithm.

[0160] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0161] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation that may be implemented in systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0162] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.

[0163] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described calibration method for a dual fisheye panoramic camera, thereby solving the technical problem of poor calibration results. Compared with the prior art, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the calibration method for a dual fisheye panoramic camera provided in the above embodiments, and will not be repeated here.

[0164] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.

Claims

1. A calibration method for a dual fisheye panoramic camera, characterized in that, The method includes: After obtaining two fisheye images from a dual fisheye panoramic camera, real-scene natural feature points in the overlapping field of view of the two fisheye images are extracted, and feature descriptors corresponding to the real-scene natural feature points are generated. The feature descriptors are matched across shots using the nearest neighbor matching algorithm, and valid matching point pairs are selected to obtain the matching results. Based on the matching results and the binocular imaging parameter calculation rules, determine the parameter calculation strategy corresponding to the matching results; Based on the parameter calculation strategy, and combined with the positional correspondence of the matching point pairs, a reprojection error model is constructed. The convergence target for parameter optimization is determined by the reprojection error model, and the camera calibration parameters are obtained by combining the nonlinear optimization algorithm.

2. The calibration method for a dual fisheye panoramic camera as described in claim 1, characterized in that, The step of extracting real-world natural feature points from the overlapping field of view of the two fisheye images obtained from the dual fisheye panoramic camera, and generating feature descriptors corresponding to the real-world natural feature points, includes: Two original fisheye images are obtained through the dual fisheye panoramic camera, and the two original fisheye images are subjected to image standardization processing to obtain the two fisheye images. Based on the initial installation pose parameters of the dual lenses and the radial distortion distribution law of the fisheye lens, an effective overlapping field of view region is defined in the two fisheye images, and an effective image block of the overlapping region is obtained by cropping. Within the effective image block of the overlapping region, the real-scene natural feature points and their corresponding pixel coordinate information are detected to obtain a set of feature point coordinates. Extract the grayscale distribution and gradient change information within a preset neighborhood of the feature points in the feature point coordinate set, and generate the feature descriptor corresponding to the real-scene natural feature point.

3. The calibration method for a dual fisheye panoramic camera as described in claim 1, characterized in that, The step of performing cross-camera feature matching on the feature descriptors according to the nearest neighbor matching algorithm, filtering valid matching point pairs, and obtaining matching results includes: The feature descriptor is retrieved point by point using the nearest neighbor matching algorithm to match candidate corresponding points for all feature points, thus obtaining an initial set of candidate matching point pairs. Based on the nearest neighbor distance and second nearest neighbor distance of the feature descriptor, the distance ratio threshold is calculated; The initial candidate matching point pair set is filtered by distance ratio, and low-confidence matching point pairs with similarity discrimination below the distance ratio threshold are removed to obtain a preliminary filtered matching point pair set; The preliminary set of matched point pairs is subjected to binocular geometric consistency verification according to the random sampling consistency algorithm. Mismatched point pairs that do not conform to the spatial correspondence are removed to obtain a set of valid matched point pairs. The number of point pairs in the set of valid matching point pairs is counted, and the number of point pairs is compared with the calibrated start threshold to obtain the matching result.

4. The calibration method for a dual fisheye panoramic camera as described in claim 1, characterized in that, The step of determining the parameter calculation strategy corresponding to the matching result based on the matching result and the binocular imaging parameter calculation rules includes: If the matching result does not meet the calibration start threshold, the overlapping field of view search area is expanded or the matching result is accumulated to obtain a set of qualified matching point pairs that meet the calibration start conditions. Based on the set of qualified matching point pairs, the pixel coordinate correspondence and confidence weight of each matching point pair are extracted to obtain a matching point coordinate mapping dataset including the confidence weight. Based on the binocular imaging parameter calculation rules and the matching point coordinate mapping dataset, construct the spatial geometric constraint relationship of the dual lenses and define the range of camera parameters participating in the calculation. Based on the spatial geometric constraints of the dual-lens camera and the camera parameter range, a parameter calculation strategy is determined with the goal of minimizing the weighted projection error.

5. The calibration method for a dual fisheye panoramic camera as described in claim 1, characterized in that, The step of constructing the reprojection error model based on the parameter calculation strategy and the positional correspondence of the matching point pairs includes: The range of parameters to be solved, weight allocation rules and optimization convergence objectives in the parameter calculation strategy are analyzed, and the actual observed pixel coordinates of all matching point pairs are combined to generate the basic dataset for modeling. Based on the fisheye imaging geometry principle, the spatial geometric constraints of dual lenses, and each pair of matching points in the modeling dataset, the reprojection calculation logic of pixel coordinates and reprojection coordinates is constructed. For each pair of matching points, a corresponding projection confidence weight is matched to the reprojection calculation logic. The initial calculation model is constructed with the weighted sum of the reprojection deviations of all matching points as the optimization objective. After verifying the parameter constraints and operational boundaries of the initial calculation model and confirming that it meets the requirements of the parameter operation strategy, the reprojection error model is output.

6. The calibration method for a dual fisheye panoramic camera as described in claim 1, characterized in that, The steps of determining the convergence objective for parameter optimization through the reprojection error model and solving for the camera calibration parameters using a nonlinear optimization algorithm include: By optimizing the convergence objective, the set of parameters to be optimized, and the matching confidence weights within the reprojection error model, the initial iteration value and step size constraint range of the nonlinear optimization algorithm are set, and the parameter iteration solution configuration is generated. Based on the parameter iterative solution configuration and the nonlinear optimization algorithm, the camera intrinsic parameters, distortion parameters and dual-lens relative extrinsic parameters are jointly iteratively calculated to obtain the global weighted projection error value. The global weighted projection error value output in each iteration is compared with a preset convergence threshold. When the global weighted projection error value is lower than the preset convergence threshold for a preset number of consecutive iterations and the fluctuation amplitude is less than the set precision, the current optimal parameter set is output. The optimal parameter set is subjected to imaging geometry consistency verification to obtain the camera calibration parameters in a complete standard format.

7. The calibration method for a dual fisheye panoramic camera as described in claim 1, characterized in that, After the steps of determining the parameter optimization convergence target through the reprojection error model and solving for the camera calibration parameters using a nonlinear optimization algorithm, the calibration method for the dual fisheye panoramic camera further includes: In response to the parameter correction command, the real-world natural feature points of the overlapping field of view of the two fisheye images are extracted, and the reprojection correspondence of the feature points is calculated according to the calibration parameter correction rules to obtain a set of projection matching point pairs. Based on the set of projection matching points, and in conjunction with the currently effective camera calibration parameters, reprojection operations are performed point by point to generate the average reprojection error, the average offset of matching points, and the continuity of the splicing seam. The average reprojection error, the average offset of the matching points, and the continuity of the splicing seam are compared with the corresponding index thresholds one by one, and the error value change trajectory of multiple consecutive frames is traced to obtain the index detection results. When any of the detected indicators in the indicator detection results exceeds the indicator threshold, or when a continuous increasing trend of error is detected in multiple consecutive frames, it is determined that the camera parameters have drifted, and the parameter drift judgment result is output.

8. The calibration method for a dual fisheye panoramic camera as described in claim 7, characterized in that, After the step of determining that camera parameters have drifted and outputting the parameter drift determination result when any of the detected indicators in the indicator detection results exceed the indicator threshold, or when a continuous increasing trend of error is detected in multiple consecutive frames, the calibration method of the dual fisheye panoramic camera further includes: Based on the parameter drift judgment result, an incremental parameter correction strategy is initiated. The camera calibration parameters are used as the initial values ​​for iteration, and the distortion parameters involved in the camera are kept unchanged. The matching point pairs of multiple consecutive frames are accumulated through a sliding time window mechanism to generate a correction-specific matching point dataset. Based on the aforementioned modified dedicated matching point dataset, with the relative extrinsic parameters of the two lenses as optimization variables, the rotation increment and translation increment of the extrinsic parameters are calculated according to the least squares optimization algorithm to obtain the extrinsic parameter increment correction amount; The relative extrinsic parameters of the dual lenses are updated according to the extrinsic parameter increment correction amount, and the updated target camera calibration parameters are loaded into the panoramic stitching processing module for subsequent image stitching operations.

9. A calibration device for a dual fisheye panoramic camera, characterized in that, The calibration device for the dual fisheye panoramic camera includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the calibration method for the dual fisheye panoramic camera as described in any one of claims 1 to 8.

10. A storage medium, characterized in that, The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the calibration method for a dual fisheye panoramic camera as described in any one of claims 1 to 8.

Images