Intelligent adaptive calibration method and system

Abstract

The invention provides an intelligent adaptive calibration method and system, and relates to the technical field of calibration, and the method comprises the steps: carrying out the spatial-temporal clustering of a real-time asynchronous event stream based on real-time motion data, carrying out the time alignment with a current RGB image frame, and generating a purification event cluster; accumulating the purification event clusters into an event grey-scale map, inputting the event grey-scale map and the current RGB image frame into a dynamic sensing network, and outputting a static mask; based on the purification event cluster, mapping the event coordinates to a coordinate system of the current RGB image frame, and performing space projection filtering by combining a static mask to form a static feature point set; constructing an objective function based on the static feature point set and performing nonlinear optimization to obtain real-time calibration parameters and calibration residual errors; the motion constraint parameters are updated based on the real-time calibration parameters, meanwhile, the static mask segmentation threshold value of the dynamic sensing network is dynamically adjusted according to the calibration residual error, and the robustness and practicability of the multi-mode visual system in the complex dynamic environment are improved.

Description

Technical Field

[0001] The present invention relates to the field of calibration technology, and in particular to an intelligent adaptive calibration method and system. Background Art

[0002] With the widespread application of intelligent perception systems in fields such as autonomous driving, robotic navigation, and augmented reality, multimodal sensor fusion technology has become a research hotspot. The collaborative calibration of visual inertial odometry (VIO), event cameras (Dynamic Vision Sensors (DVS), and RGB cameras is particularly critical. Traditional calibration methods rely primarily on static scenes and manually designed calibration plates, making them difficult to adapt to the real-time application requirements of dynamic and complex environments.

[0003] Currently, mainstream calibration methods typically rely on feature point matching, such as using a checkerboard or dot array to estimate camera internal and external parameters. However, these methods are prone to calibration failure in the presence of motion blur, varying illumination, or missing textures. Furthermore, event cameras, as novel asynchronous visual sensors, output a sparse event stream with high temporal resolution. Traditional synchronous image processing methods struggle to effectively process this data, limiting their potential for real-time calibration.

[0004] Therefore, it is necessary to provide an intelligent adaptive calibration method and system to solve the above technical problems. Summary of the Invention

[0005] To solve the above technical problems, the present invention provides an intelligent adaptive calibration method and system. By fusing the spatiotemporal information of asynchronous event streams and RGB images, and combining dynamic perception with nonlinear optimization technology, high-precision and adaptive online calibration is achieved, which significantly improves the robustness and practicality of multimodal vision systems in complex dynamic environments. The present invention provides an intelligent adaptive calibration method, which includes the following steps: Based on real-time motion data, the motion compensation module is used to perform spatiotemporal clustering on the real-time asynchronous event stream and time-align it with the current RGB image frame to generate spatiotemporally aligned purified event clusters. Accumulate the purified event clusters into an event grayscale image, and input it into a dynamic perception network together with the current RGB image frame to output a static mask; Based on the purified event cluster, the event coordinates are mapped to the coordinate system of the current RGB image frame using a spatiotemporal projection model constructed based on the initial calibration parameters, and spatial projection filtering is performed in combination with the static mask to form a static feature point set; Constructing an objective function based on the static feature point set, and performing nonlinear optimization on the objective function to obtain real-time calibration parameters and calibration residuals; The motion constraint parameters in the spatiotemporal projection model are updated based on the real-time calibration parameters, and the static mask segmentation threshold of the dynamic perception network is dynamically adjusted according to the calibration residual.

[0006] Preferably, the generation process of the purification event cluster specifically includes: Acquire six-degree-of-freedom motion data in real time, wherein the six degrees of freedom include three-axis angular velocity and three-axis acceleration; Performing spatiotemporal density clustering on the asynchronous event stream generated by the dynamic visual sensor in a preset time window, merging events that satisfy spatial neighborhood and temporal correlation to form event clusters, calculating a rigid body motion transformation matrix based on the motion data, and performing motion compensation on the event clusters; The motion-compensated event clusters are temporally aligned with the current RGB image frame to generate spatiotemporally aligned purified event clusters.

[0007] Preferably, the output process of the static mask specifically includes: Based on the purified event cluster, the number of events for each pixel in the event cluster is subjected to exponential decay weighted accumulation with a preset time window as a period, to generate an event grayscale image consistent with the current RGB image frame resolution; The dual-branch encoder of the dynamic perception network extracts the texture features of the RGB image frame and the motion features of the event grayscale image respectively; The cross-modal attention mechanism of the dynamic perception network is used to calculate the spatial correlation weight matrix of the texture features and the motion features, and the features are fused according to the spatial correlation weight matrix to output a static mask and its binary identification area.

[0008] Preferably, the process of forming the static feature point set specifically includes: Based on the camera intrinsic parameter matrix, extrinsic parameter rotation matrix, and translation vector in the initial calibration parameters, the spatiotemporal projection model is constructed, and its mapping relationship for event coordinate transformation is: in, is the two-dimensional coordinate of the coordinate system projected to the current RGB image frame, To purify the three-dimensional event coordinates of the event cluster, is the extrinsic rotation matrix, is the translation vector, is the camera intrinsic parameter matrix; Matching the projected two-dimensional coordinates with the binary identification area of the static mask, and retaining event points that meet the preset filtering conditions based on the matching results; FAST corner detection is performed on event points that meet the preset filtering conditions to form a sparsely distributed static feature point set.

[0009] Preferably, the preset filtering condition is that the event point projection coordinates are located in an area marked as a valid state within the binary identification area of the static mask.

[0010] Preferably, the objective function construction process includes: The objective function is constructed using the reprojection error between the coordinates of the sparsely distributed corner points in the static feature point set and the two-dimensional projection coordinates corresponding to the current RGB image frame; wherein the reprojection error is defined as the difference between the predicted two-dimensional coordinates of each static feature point after conversion by the spatiotemporal projection model and the actually observed two-dimensional coordinates.

[0011] Preferably, the nonlinear optimization process includes: The extrinsic rotation matrix and translation vector in the initial calibration parameters are iteratively adjusted by using the LM algorithm to minimize the objective function and obtain real-time calibration parameters.

[0012] Preferably, the process of obtaining the calibration residual includes: The calibration residual corresponding to each static feature point is calculated based on the obtained real-time calibration parameters.

[0013] Preferably, updating the motion constraint parameters in the spatiotemporal projection model based on the real-time calibration parameters and dynamically adjusting the static mask segmentation threshold of the dynamic perception network according to the calibration residual includes: Updating the external parameter rotation matrix and translation vector in the real-time calibration parameters into the space-time projection model, replacing the initial calibration parameters, and recalculating the event coordinate mapping relationship based on the updated space-time projection model; Based on the calibration residual, the static mask segmentation threshold of the dynamic perception network is dynamically adjusted according to preset rules.

[0014] The present invention also provides an intelligent adaptive calibration system for executing an intelligent adaptive calibration method, the system comprising: The spatiotemporal alignment module is used to perform spatiotemporal clustering of the real-time asynchronous event stream based on real-time motion data using the motion compensation module, and to perform temporal alignment with the current RGB image frame to generate spatiotemporally aligned purified event clusters; A static mask generation module is used to accumulate the purified event clusters into an event grayscale image, input it into a dynamic perception network together with the current RGB image frame, and output a static mask; A static feature point extraction module is used to map the event coordinates to the coordinate system of the current RGB image frame based on the purification event cluster using a spatiotemporal projection model constructed based on the initial calibration parameters, and perform spatial projection filtering in combination with the static mask to form a static feature point set; a calibration parameter optimization module, configured to construct an objective function based on the static feature point set, and perform nonlinear optimization on the objective function to obtain real-time calibration parameters and calibration residuals; A closed-loop parameter updating module is used to update the motion constraint parameters in the spatiotemporal projection model based on the real-time calibration parameters, and dynamically adjust the static mask segmentation threshold of the dynamic perception network according to the calibration residual.

[0015] Compared with related technologies, the intelligent adaptive calibration method and system provided by the present invention have the following beneficial effects: This method introduces a motion compensation mechanism to perform spatiotemporal clustering and alignment of asynchronous event streams, effectively addressing the calibration errors caused by sensor asynchrony and motion blur in dynamic scenes. Furthermore, by combining the dual-modal information of event grayscale images and RGB image frames, a dynamic perception network is used to extract static masks, effectively suppressing non-static interference regions and improving the robustness of feature extraction.

[0016] This method uses a spatiotemporal projection model to map event data to an image coordinate system. This method, combined with a static mask for spatial filtering, constructs a sparse and stable set of static feature points, providing a high-quality data foundation for subsequent parameter optimization. By constructing an objective function based on reprojection error and employing a nonlinear optimization algorithm for parameter estimation, it enables real-time updating of camera internal and external parameters, improving the system's adaptability and calibration accuracy.

[0017] In addition, the present invention dynamically adjusts the motion constraint parameters and segmentation thresholds in the model through a calibration residual feedback mechanism, so that the system can maintain good stability and generalization ability under different motion states and environmental conditions. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 A flow chart of an intelligent adaptive calibration method provided by the present invention; Figure 2 This is a module structure diagram of an intelligent adaptive calibration method provided by the present invention. DETAILED DESCRIPTION

[0019] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all of the structures. Furthermore, the embodiments of the present invention and the features of the embodiments may be combined with one another unless there is a conflict.

[0020] It should also be noted that, for ease of description, only portions relevant to the present invention are shown in the accompanying drawings, rather than all of the contents. Before discussing the exemplary embodiments in more detail, it should be noted that some exemplary embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the various operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently, or simultaneously. In addition, the order of the various operations can be rearranged. The process can be terminated when its operations are completed, but may also have additional steps not included in the accompanying drawings. The process can correspond to a method, function, procedure, subroutine, subprogram, etc.

[0021] Example 1 The present invention provides an intelligent adaptive calibration method, referring to Figure 1 As shown, the method includes the following steps: S1: Based on real-time motion data, the motion compensation module is used to perform spatiotemporal clustering on the real-time asynchronous event stream and temporally align it with the current RGB image frame to generate spatiotemporally aligned purified event clusters.

[0022] Specifically, in step S1, the generation process of the purification event cluster specifically includes: S11: Acquire six-degree-of-freedom motion data in real time, wherein the six degrees of freedom include three-axis angular velocity and three-axis acceleration.

[0023] In this embodiment, motion data is collected in real time by a 6-DOF inertial measurement unit (IMU) integrated into the device, with a sampling frequency of 1kHz. Three-axis angular velocity data is acquired via a gyroscope with a range of ±2000° / s; three-axis acceleration data is acquired via an accelerometer with a range of ±16g. The raw data is filtered with a sliding window average (window length 50ms) and attitude is calculated using a quaternion complementary filtering algorithm, outputting the device's three-dimensional rotation angles (roll, pitch, yaw) and displacement vector ( ) as rigid body motion parameters and are output to the motion compensation module at a 20ms cycle. The rotation angle is converted to the rotation matrix R using the Rodrigues formula, and the displacement vector is calculated by quadratic integration of the acceleration to obtain the translation vector T. Together, these form the basic parameters for rigid body motion transformation.

[0024] S12: Perform spatiotemporal density clustering on the asynchronous event stream generated by the dynamic visual sensor in a preset time window, merge events that satisfy spatial neighborhood and temporal correlation to form event clusters, calculate the rigid body motion transformation matrix based on the motion data, and perform motion compensation on the event clusters.

[0025] In this embodiment, the asynchronous event stream output by the dynamic vision sensor is stored as a coordinate-time series data packet with a microsecond timestamp. The system performs spatiotemporal density clustering on the event stream with a fixed time window of 50ms: an improved DBSCAN algorithm is used with a spatial neighborhood radius of 5 pixels and a temporal correlation threshold of 2ms. Based on the rotation matrix R and translation vector T obtained in step S11, the rigid body motion transformation matrix is constructed through the exponential mapping of the Lie group SE (3) , its specific form is a 4×4 homogeneous matrix . The three-dimensional coordinates of each event cluster , apply the inverse of the transformation matrix Perform motion compensation and calculate the compensated coordinates , eliminating the coordinate offset caused by device motion. The compensated event coordinates are resampled to the standard grid through bilinear interpolation.

[0026] S13: Temporally aligning the motion-compensated event cluster with the current RGB image frame to generate a spatiotemporally aligned purified event cluster.

[0027] In this example, the motion-compensated event cluster is time-aligned with the current frame captured by a global shutter RGB camera. Based on the midpoint exposure time t_frame of the RGB frame, events within the time window [t_frame - 5ms, t_frame + 5ms] are selected, and cubic spline interpolation is used to achieve sub-millisecond time alignment (deviation ≤ 0.1ms).

[0028] The aligned event clusters are then opened using a morphological opening operation (3×3 rectangular kernel) to remove isolated noise points, and low-confidence regions are filtered using a spatial density threshold (≥10 events / mm²). This results in a purified event cluster that is spatially aligned and stored as a . , precise timestamp t (microsecond level), polarity A structured array of .

[0029] S2: Accumulate the purified event clusters into an event grayscale image, and input it together with the current RGB image frame into a dynamic perception network to output a static mask.

[0030] Specifically, in step S2, the static mask output process specifically includes: S21: Based on the purified event cluster, the number of events of each pixel in the event cluster is subjected to exponential decay weighted accumulation with a preset time window as a period, and an event grayscale image consistent with the current RGB image frame resolution is generated.

[0031] In this embodiment, based on the spatiotemporally aligned cleansed event clusters generated in step S13, event data is accumulated with a fixed time window of 50ms (consistent with the spatiotemporal clustering window in step S1). The number of events per pixel is calculated using an exponentially decaying weighting function: in, is the base of natural logarithms, Represents the coordinates in the output event grayscale image The pixel value at Indicates the index of all events in the current time window. is the attenuation coefficient (adapted to 50ms window), which controls the weight attenuation speed of historical events. Indicates the end timestamp of the current time window. For the The timestamp of the event, Indicates event polarity, and the accumulated grayscale value is linearly normalized to Range, generate event grayscale image aligned with RGB image frame resolution, and perform bilateral filtering on grayscale image to eliminate sensor noise (spatial standard deviation , intensity standard deviation ).

[0032] S22: The texture features of the RGB image frame and the motion features of the event grayscale image are extracted respectively through the dual-branch encoder of the dynamic perception network.

[0033] In this embodiment, the dynamic perception network adopts a dual-branch ResNet-18 architecture: RGB branch: After the input RGB image is normalized (mean [0.485, 0.456, 0.406], standard deviation [0.229, 0.224, 0.225]), the third layer feature map (size 256×14×14) is extracted through the pre-trained ResNet-18 to capture texture and semantic information.

[0034] Event branch: The event grayscale image undergoes adaptive histogram equalization and is then fed into a modified ResNet-18 (with the first-layer convolution kernel adjusted to a single channel). A third-layer motion feature map (size 256×14×14) is simultaneously extracted to encode dynamic changes. L2 regularization is applied to the dual-branch feature maps to ensure feature scale consistency.

[0035] S23: Utilize the cross-modal attention mechanism of the dynamic perception network to calculate the spatial correlation weight matrix of the texture features and the motion features, fuse the features according to the spatial correlation weight matrix, and output a static mask and its binary identification area.

[0036] In this embodiment, the cross-modal attention mechanism calculates the spatial correlation weight matrix as follows: in, The query vector is reshaped from the RGB feature map, and the 256-dimensional feature is projected to 64 dimensions through 1×1 convolution, where: H×W=14×14 is the feature map space size and C=64 is the feature dimension after projection; The feature map of the event branch is reshaped into a key vector, which is also projected to 64 dimensions through 1×1 convolution. is the feature dimension scaling factor, used to stabilize gradient calculation, is the matrix transpose operator, The exponential function is row-normalized to ensure that the weights of each row sum to 1.

[0037] Fusion features according to Generate, after 3×3 convolution layer and Sigmoid activation, output static mask. Binary identification area is generated by adaptive threshold segmentation: in, The final generated binary static mask is at pixel coordinates The value at , which is 0 (invalid area) or 1 (valid area). This mask is used to distinguish the static background from the dynamic objects in the scene in the subsequent steps. is the basic segmentation threshold, equal to 0.5, is the standard deviation of the features within the 5×5 window, is the local adaptive coefficient (initial experimental optimization value, which needs to be dynamically adjusted later), which is equal to 0.2. The final mask is filled with holes through morphological closing operation (5×5 ellipse kernel), and the noise area with an area less than 100 pixels is removed through connected domain analysis.

[0038] S3: Based on the purified event cluster, the event coordinates are mapped to the coordinate system of the current RGB image frame using a spatiotemporal projection model constructed based on the initial calibration parameters, and spatial projection filtering is performed in combination with the static mask to form a static feature point set.

[0039] Specifically, in step S3, the process of forming the static feature point set includes: S31: Based on the camera intrinsic parameter matrix, extrinsic parameter rotation matrix, and translation vector in the initial calibration parameters, the spatiotemporal projection model is constructed. The mapping relationship for event coordinate conversion is: in, is the two-dimensional coordinate of the coordinate system projected to the current RGB image frame, To purify the three-dimensional event coordinates of the event cluster, is the extrinsic rotation matrix, is the translation vector, is the camera intrinsic parameter matrix.

[0040] In this embodiment, a spatiotemporal projection model is constructed based on initial calibration parameters. The camera intrinsic parameter matrix is pre-calibrated using the Zhang calibration method and includes the focal length and principal point coordinate parameters. The focal length is set to 520.5 pixels and 521.2 pixels, respectively, and the principal point coordinates are located near the center of the image (320.1 pixels on the horizontal axis and 240.8 pixels on the vertical axis). The initial value of the extrinsic rotation matrix is obtained by joint calibration of the IMU and the camera. The Euler angles (roll angle 0.1°, pitch angle 0.05°, yaw angle -0.2°) are converted into a 3×3 rotation matrix, and the initial value of the translation vector is set to [0.2 meters, -0.1 meters, 1.5 meters].

[0041] The mapping process of event coordinates is as follows: the three-dimensional spatiotemporal coordinates of the event (including pixel position and microsecond timestamp) are converted to the current RGB image coordinate system through the intrinsic parameter matrix, rotation matrix and translation vector, and the radial distortion is corrected by applying the Brown-Conrady distortion model (distortion coefficient k1 is -0.12 and k2 is 0.03), finally obtaining the projection coordinates with sub-pixel accuracy.

[0042] S32: matching the projected two-dimensional coordinates with the binary identification area of the static mask, and retaining event points that meet preset filtering conditions based on the matching results.

[0043] In this embodiment, if the projected coordinates of the event point are located within the valid area where the static mask flag is 1, the event point is retained.

[0044] For non-integer coordinates, bilinear interpolation is used to calculate the weighted values of the four adjacent pixels of the mask (the weight is determined by the proportion of the decimal part). If the interpolated mask value is ≥ 0.5, it is considered valid.

[0045] S33: Perform FAST corner point detection on event points that meet preset filtering conditions to form a sparsely distributed static feature point set.

[0046] In this example, FAST corner detection and post-processing are performed on the filtered event points. Detection parameters are set as follows: a grayscale difference threshold of 20 (range 0-255) and a detection window of 7×7 pixels. Non-maximum suppression is used to retain local response maxima (3×3 neighborhood comparison), and unstable feature points within 10 pixels of the image edge are removed.

[0047] The detected corner points are spatially deduplicated (if the distance between two points is less than 5 pixels, they are merged), ultimately generating a sparsely distributed static feature point set. Each feature point records its projected coordinates, the original event cluster ID, and the initial residual value (preset to 0.5 pixels).

[0048] S4: constructing an objective function based on the static feature point set, and performing nonlinear optimization on the objective function to obtain real-time calibration parameters and calibration residuals.

[0049] Specifically, in step S4, the objective function construction process includes: The objective function is constructed using the reprojection error between the coordinates of the sparsely distributed corner points in the static feature point set and the two-dimensional projection coordinates corresponding to the current RGB image frame; wherein the reprojection error is defined as the difference between the predicted two-dimensional coordinates of each static feature point after conversion by the spatiotemporal projection model and the actually observed two-dimensional coordinates.

[0050] In this embodiment, all static feature points are traversed, and the Euclidean distance between each point's predicted 2D coordinates, transformed using the spatiotemporal projection model, and the actual observed coordinates in the RGB image is calculated as the reprojection error, with the error unit being pixels. To suppress outlier interference, the Huber robust kernel function is used to weight the error: when the reprojection error of a single point is less than or equal to 1.5 pixels, the squared error is used directly; when the error exceeds 1.5 pixels, a linear loss function is used instead (the error value is multiplied by twice the threshold value and then subtracted from half the square of the threshold value). IMU motion prior constraints are also introduced, using the Lie algebraic difference between the extrinsic rotation matrix and the rotation calculated by the IMU pre-integration as a regularization term. An experimentally tuned weight coefficient of 0.3 balances reprojection error and sensor consistency. The weight value is dynamically fine-tuned within a ±0.05 range based on the confidence level of the IMU data before each frame optimization.

[0051] Specifically, in step S4, the nonlinear optimization process includes: The extrinsic rotation matrix and translation vector in the initial calibration parameters are iteratively adjusted by using the LM algorithm to minimize the objective function and obtain real-time calibration parameters.

[0052] In this embodiment, the optimization variables are initialized to the Euler angles (roll, pitch, and yaw) and the three-dimensional components of the translation vector of the extrinsic rotation matrix. The initial values are taken from the calibration parameters in step S3. An iterative optimization is performed using the LM algorithm (Levenberg-Marquardt algorithm), with an initial damping factor of 0.001 and a maximum number of 50 iterations. Automatic differentiation is employed to calculate the Jacobian matrix in each iteration, using PyTorch's autograd module to differentiate the Lie algebraic parameters of the rotation matrix to avoid gimbal lock. After each parameter update, the orthogonality of the rotation matrix is enforced through Gram-Schmidt orthogonalization, ensuring that its determinant is between 0.999 and 1.001. The optimization is terminated when the residual error decreases by less than 0.1% over three consecutive iterations. The optimized extrinsic rotation matrix and translation vector are ultimately output as real-time calibration parameters.

[0053] Specifically, in step S4, the process of obtaining the calibration residual includes: The calibration residual corresponding to each static feature point is calculated based on the obtained real-time calibration parameters.

[0054] In this embodiment, the optimized external parameter parameters are substituted into the spatiotemporal projection model, the projection coordinates of all static feature points are recalculated, and the updated reprojection error is obtained. The median absolute deviation method is used to screen out abnormal points: the median and absolute deviation of all errors are calculated, and points whose errors exceed the median plus or minus three times the absolute deviation range are eliminated. The root mean square value (RMS) of the retained valid error value is calculated as the calibration residual output of the current frame. At the same time, a ring buffer is established to store the residual data of the most recent 10 frames, and the moving average and variance of the residual in the window are statistically calculated for subsequent dynamic threshold adjustment. The residual history record of each feature point is saved to its associated event cluster ID. When the residual of a certain ID exceeds 2.5 pixels for three consecutive frames, the weight downgrade mechanism of the cluster event point is triggered, and its confidence coefficient is reduced by 50% in the next frame processing.

[0055] S5: updating the motion constraint parameters in the spatiotemporal projection model based on the real-time calibration parameters, and dynamically adjusting the static mask segmentation threshold of the dynamic perception network according to the calibration residual.

[0056] Specifically, step S5 includes the following steps: S51: updating the external parameter rotation matrix and translation vector in the real-time calibration parameters into the space-time projection model, replacing the initial calibration parameters, and recalculating the event coordinate mapping relationship based on the updated space-time projection model.

[0057] In this embodiment, the parameter update process is first executed. The real-time calibration parameters (Euler angles and translation vectors of the extrinsic rotation matrix) obtained by optimization in step S4 are replaced with the original parameters in the spatiotemporal projection model in an atomic operation to ensure data consistency during multi-threaded processing.

[0058] After an update, the coordinate mapping is immediately recalculated: the 3D coordinates of all event clusters in the current frame are traversed, and the projection operation is re-performed using the new extrinsic parameter matrix. A double buffering mechanism is used to parallelize the calculation of the old and new parameters for three frames to achieve a smooth transition. After each parameter update, orthogonality verification is performed, using Gram-Schmidt orthogonalization to enforce the orthogonality of the rotation matrix. If the correction exceeds 0.01 radians, an exception alarm is triggered and the parameters are rolled back to the previous valid state. The updated projection coordinate data is written to a shared memory pool for subsequent processing modules to call in a pipelined manner.

[0059] S52: Based on the calibration residual, dynamically adjust the static mask segmentation threshold of the dynamic perception network according to a preset rule.

[0060] In this embodiment, the mask threshold is then dynamically adjusted. Based on the calibration residual data output in step S4, the residual root mean square (RMS) of the last 10 frames is extracted from the ring buffer to construct a sliding window statistic. When the residual mean in the window exceeds 1.2 pixels, the basic segmentation threshold is increased by 0.03 for every 0.1 pixel residual increment, with a maximum of no more than 0.65; when the residual mean is less than 0.8 pixels, the threshold is reduced by 0.02 for every 0.1 pixel decrement, with a minimum of no less than 0.35. At the same time, the local adaptive coefficient is adjusted according to the residual variance: when the variance is greater than 0.5, the coefficient Increased to 0.25 to enhance regional adaptability, the coefficient is less than 0.2 The threshold value was reduced to 0.15 to enhance global consistency. The updated threshold parameters were injected into the dynamic perception network via a dynamic link library hot update mechanism, taking effect in real time during network forward inference, while retaining a three-frame delay for smooth weight transitions. When a sudden increase in the residual error of a single frame is detected that exceeds three standard deviations of the historical mean, an emergency recovery mechanism is activated, temporarily freezing the threshold adjustment and enabling the use of backup calibration parameters.

[0061] Example 2 The present invention also provides an intelligent adaptive calibration system for executing an intelligent adaptive calibration method, referring to Figure 2 As shown, the system includes: The spatiotemporal alignment module 100 is used to perform spatiotemporal clustering on the real-time asynchronous event stream based on real-time motion data using a motion compensation module, and to perform time alignment with the current RGB image frame to generate spatiotemporally aligned purified event clusters.

[0062] The static mask generation module 200 is used to accumulate the purified event clusters into an event grayscale image, input the event grayscale image together with the current RGB image frame into a dynamic perception network, and output a static mask.

[0063] The static feature point extraction module 300 is used to map the event coordinates to the coordinate system of the current RGB image frame based on the purification event cluster using a spatiotemporal projection model constructed based on the initial calibration parameters, and perform spatial projection filtering in combination with the static mask to form a static feature point set.

[0064] The calibration parameter optimization module 400 is used to construct an objective function based on the static feature point set, and perform nonlinear optimization on the objective function to obtain real-time calibration parameters and calibration residuals.

[0065] The closed-loop parameter updating module 500 is used to update the motion constraint parameters in the spatiotemporal projection model based on the real-time calibration parameters, and dynamically adjust the static mask segmentation threshold of the dynamic perception network according to the calibration residual.

[0066] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0067] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program. The program can be stored in a computer-readable storage medium, including a read-only memory (ROM), a random access memory (RAM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM), or other optical disc storage, magnetic disk storage, or magnetic tape storage, or any other computer-readable medium capable of carrying or storing data.

[0068] It should also be noted that the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, commodity, or apparatus that includes a series of elements includes not only those elements but also other elements not explicitly listed, or includes elements inherent to such process, method, commodity, or apparatus. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the process, method, commodity, or apparatus that includes the element.

Claims

1. An intelligent adaptive calibration method, characterized in that: The method comprises the following steps: Based on real-time motion data, the motion compensation module is used to perform spatiotemporal clustering on the real-time asynchronous event stream and time-align it with the current RGB image frame to generate spatiotemporally aligned purified event clusters. Accumulate the purified event clusters into an event grayscale image, and input it into a dynamic perception network together with the current RGB image frame to output a static mask; Based on the purified event cluster, the event coordinates are mapped to the coordinate system of the current RGB image frame using a spatiotemporal projection model constructed based on the initial calibration parameters, and spatial projection filtering is performed in combination with the static mask to form a static feature point set; Constructing an objective function based on the static feature point set, and performing nonlinear optimization on the objective function to obtain real-time calibration parameters and calibration residuals; The motion constraint parameters in the spatiotemporal projection model are updated based on the real-time calibration parameters, and the static mask segmentation threshold of the dynamic perception network is dynamically adjusted according to the calibration residual.

2. The intelligent adaptive calibration method according to claim 1, characterized in that: The generation process of the purification event cluster specifically includes: Acquire six-degree-of-freedom motion data in real time, wherein the six degrees of freedom include three-axis angular velocity and three-axis acceleration; Performing spatiotemporal density clustering on the asynchronous event stream generated by the dynamic visual sensor in a preset time window, merging events that satisfy spatial neighborhood and temporal correlation to form event clusters, calculating a rigid body motion transformation matrix based on the motion data, and performing motion compensation on the event clusters; The motion-compensated event clusters are temporally aligned with the current RGB image frame to generate spatiotemporally aligned purified event clusters.

3. The intelligent adaptive calibration method according to claim 2, characterized in that: The output process of the static mask specifically includes: Based on the purified event cluster, the number of events for each pixel in the event cluster is subjected to exponential decay weighted accumulation with a preset time window as a period, to generate an event grayscale image consistent with the current RGB image frame resolution; The dual-branch encoder of the dynamic perception network extracts the texture features of the RGB image frame and the motion features of the event grayscale image respectively; The cross-modal attention mechanism of the dynamic perception network is used to calculate the spatial correlation weight matrix of the texture features and the motion features, and the features are fused according to the spatial correlation weight matrix to output a static mask and its binary identification area.

4. The intelligent adaptive calibration method according to claim 3, characterized in that: The process of forming the static feature point set specifically includes: Based on the camera intrinsic parameter matrix, extrinsic parameter rotation matrix, and translation vector in the initial calibration parameters, the spatiotemporal projection model is constructed, and its mapping relationship for event coordinate transformation is: in, is the two-dimensional coordinate of the coordinate system projected to the current RGB image frame, To purify the three-dimensional event coordinates of the event cluster, is the extrinsic rotation matrix, is the translation vector, is the camera intrinsic parameter matrix; Matching the projected two-dimensional coordinates with the binary identification area of the static mask, and retaining event points that meet the preset filtering conditions based on the matching results; FAST corner detection is performed on event points that meet the preset filtering conditions to form a sparsely distributed static feature point set.

5. The intelligent adaptive calibration method according to claim 4, characterized in that: The preset filtering condition is that the event point projection coordinates are located in an area marked as a valid state in the binary identification area of the static mask.

6. The intelligent adaptive calibration method according to claim 5, characterized in that: The objective function construction process includes: The objective function is constructed using the reprojection error between the coordinates of the sparsely distributed corner points in the static feature point set and the two-dimensional projection coordinates corresponding to the current RGB image frame; wherein the reprojection error is defined as the difference between the predicted two-dimensional coordinates of each static feature point after conversion by the spatiotemporal projection model and the actually observed two-dimensional coordinates.

7. The intelligent adaptive calibration method according to claim 6, characterized in that: The nonlinear optimization process includes: The extrinsic rotation matrix and translation vector in the initial calibration parameters are iteratively adjusted by using the LM algorithm to minimize the objective function and obtain real-time calibration parameters.

8. The intelligent adaptive calibration method according to claim 7, characterized in that: The process of obtaining the calibration residual includes: The calibration residual corresponding to each static feature point is calculated based on the obtained real-time calibration parameters.

9. The intelligent adaptive calibration method according to claim 8, characterized in that: The updating of motion constraint parameters in the spatiotemporal projection model based on the real-time calibration parameters and the dynamic adjustment of the static mask segmentation threshold of the dynamic perception network according to the calibration residual include: Updating the external parameter rotation matrix and translation vector in the real-time calibration parameters into the space-time projection model, replacing the initial calibration parameters, and recalculating the event coordinate mapping relationship based on the updated space-time projection model; Based on the calibration residual, the static mask segmentation threshold of the dynamic perception network is dynamically adjusted according to preset rules.

10. An intelligent adaptive calibration system, configured to execute an intelligent adaptive calibration method according to any one of claims 1 to 9, characterized in that: The system comprises: The spatiotemporal alignment module is used to perform spatiotemporal clustering of the real-time asynchronous event stream based on real-time motion data using the motion compensation module, and to perform temporal alignment with the current RGB image frame to generate spatiotemporally aligned purified event clusters; A static mask generation module is used to accumulate the purified event clusters into an event grayscale image, input it into a dynamic perception network together with the current RGB image frame, and output a static mask; A static feature point extraction module is used to map the event coordinates to the coordinate system of the current RGB image frame based on the purification event cluster using a spatiotemporal projection model constructed based on the initial calibration parameters, and perform spatial projection filtering in combination with the static mask to form a static feature point set; a calibration parameter optimization module, configured to construct an objective function based on the static feature point set, and perform nonlinear optimization on the objective function to obtain real-time calibration parameters and calibration residuals; A closed-loop parameter updating module is used to update the motion constraint parameters in the spatiotemporal projection model based on the real-time calibration parameters, and dynamically adjust the static mask segmentation threshold of the dynamic perception network according to the calibration residual.

Images