A multi-camera collaborative calibration method and application for dynamic vision sensors

Abstract

The present invention provides a multi-camera collaborative calibration method and application suitable for dynamic vision sensors, comprising the following steps: S1, obtaining sub-pixel corner point set information; S2, optimizing camera parameters to minimize reprojection errors, thereby improving the accuracy of the camera model; S3, using a random sampling consensus algorithm to calculate a first transformation matrix and a second transformation matrix based on the sub-pixel corner point set information, wherein the first matrix is ​​a transformation matrix of a frame camera relative to the origin of a reference coordinate system, and the second matrix is ​​a transformation matrix of an event camera relative to the origin of the reference coordinate system; S4, solving the transformation matrix from the event camera pixel space to the frame camera pixel space based on the obtained first transformation matrix and second transformation matrix. The present invention provides a multi-camera collaborative calibration method and application suitable for dynamic vision sensors, which can be applied to the calibration between 1-multiple event cameras and 1-multiple frame cameras, and cameras at different angles and positions.

Description

Technical Field

[0001] The present invention belongs to the technical field of camera calibration, and in particular relates to a multi-camera efficient collaborative calibration method and application suitable for dynamic vision sensors. Background Art

[0002] With the development of the Internet, visual sensors are widely used in robot vision, self-driving cars, virtual reality and augmented reality, medical imaging, security monitoring and other fields because of their advantages of quickly collecting and transmitting data. Traditional camera technology has difficulties in capturing fast-moving or high-dynamic range scenes because they capture the entire screen at a fixed frame rate and the frame rate is limited to less than 100 frames. To solve these problems, researchers began to explore the use of event-driven sensors to achieve high-speed, high-sensitivity and low-latency image capture. The development of this technology is due to the advancement of event sensor technology. Dynamic vision sensors can capture changes in light intensity at a time granularity of microseconds instead of capturing the entire screen at a fixed frame rate. Compared with traditional vision sensors, dynamic vision sensors have the characteristics of ultra-high speed, asynchronous operation, and high dynamic range.

[0003] With the development of dynamic vision sensor technology, related event processing algorithms have also been widely studied and applied. These algorithms are usually based on machine learning and artificial intelligence technologies, which can process event data in real time and provide efficient descriptions of scenes. This development provides strong support for the application of event camera technology in computer vision, robotics, self-driving cars, virtual reality, and augmented reality. Therefore, the rise of dynamic vision sensors is not only due to the challenge to traditional technologies, but also due to the progress of event sensor technology and event processing algorithms. This technological development has brought new possibilities to various fields, especially in applications that require high-speed, high-sensitivity, and low-latency sensors. With the continuous advancement of related technologies, dynamic vision sensors are expected to play an increasingly important role in the future.

[0004] Before 2016, event cameras were often used together with traditional cameras to complete tasks because they could not complete a specific task independently. After that, the research on event cameras continued to improve, gradually abandoning the assistance of other sensors, and were able to independently complete tasks such as feature tracking, mapping, optical flow estimation, and target detection. Due to the limitations of event camera data such as incomplete information, integration with traditional cameras is still very important.

[0005] On August 11, 2016, combined detection and tracking based on frames and events was published in [C / 0L] / / 2016IEEEInternational SymposiumonCircuitsandSystems(ISCAS).2016:2511-2514. After detecting the target object in a frame image, the moving direction of the target object is predicted by the data of the event camera, so that the target detection is performed in the predicted area when the next frame image arrives, thereby improving the speed and accuracy of detection. In order to effectively predict the position of the target object in the next frame, accurate spatial alignment and time synchronization between the frame camera and the event camera are required. If the calibration is insufficient, it may lead to inconsistency between the frame and event data in time and space, which in turn affects the accuracy of target detection and tracking.

[0006] On May 20, 2019, hybrid frame / event-driven fast pedestrian detection was published in the 2019 International Conference on Robotics and Automation (ICRA) May 2019 Pages 8332-8338. Two target detection networks were trained to detect pedestrians in traditional images and event frames respectively, and the confidence maps were generated and fused to obtain the final pedestrian detection results, which improved the accuracy of pedestrian detection in dim environments. In this method, the precise spatial alignment of traditional images and event data is crucial to the accuracy of pedestrian detection. If the camera calibration is inaccurate, it will cause data alignment deviation, thereby affecting the accuracy and reliability of the confidence map after fusion. This alignment problem may cause false detection or missed detection in the pedestrian detection system in practical applications.

[0007] On July 26, 2021, standard and event camera fusion for feature tracking was presented at the 2021 International Conference on Machine Vision and Applications (ICMVA 2021). It uses images from traditional cameras to extract features, then tracks them through event streams and makes corrections when new images arrive, making full use of traditional images and achieving certain advantages over pure event stream tracking. In order to effectively fuse the two types of data, the traditional images and event data must be accurately aligned. If the calibration is not accurate enough, feature positions will be misaligned, which is critical for tracking using event data and making adjustments to new images. The accuracy of spatial alignment directly affects the continuity and accuracy of tracking.

[0008] On August 5, 2020, high dynamic range imaging guided by neuromorphic cameras was published in the 2020 IEEE / CVFConference on Computer Vision and Pattern Recognition (CVPR). The U-Net structure network was used to fuse the event camera data stream and traditional images to obtain HDR images, solving the problem of low dynamic range of images taken by traditional cameras. After fusion, the image details of the reflective part of the car body were supplemented. However, in the process of video image synthesis, accurate spatial alignment is essential to ensure the effective combination of traditional images and event data. Any error in calibration will affect the accuracy of alignment, which will not only reduce the quality of image fusion, but also may damage its realism. Especially when dealing with dynamic scenes, the accuracy of alignment becomes particularly critical.

[0009] In some scenarios where the event camera and the traditional frame camera RGB multi-camera system need to be jointly worked, the event camera and the traditional frame camera need to be jointly calibrated first to ensure that the collected data is synchronized in space, thereby providing an accurate basis for subsequent data processing and analysis. The quality of the joint calibration directly affects the accuracy and reliability of subsequent data processing and analysis.

[0010] At present, the calibration method suitable for event cameras is relatively complex, with cumbersome steps and requires a lot of computing resources. In terms of implementation difficulty, it is highly dependent on high-precision hardware devices or high-precision complex calibration plates. During the calibration process, the relative motion between the camera and the calibration plate indirectly introduces errors, affecting the accuracy of the final calibration results.

[0011] Therefore, how to solve the problems of high complexity, high dependence and error introduction of event camera calibration, provide a method for achieving high stability and high precision of calibration results, and efficiently complete the joint calibration of event camera multi-camera systems, is a technical problem that needs to be urgently solved by technical personnel in this field. Summary of the invention

[0012] The first objective of the present invention is to provide a multi-camera collaborative calibration method suitable for dynamic vision sensors, in order to address the problem of low multi-camera calibration accuracy of event cameras and frame cameras in the prior art.

[0013] To this end, the above-mentioned purpose of the present invention is achieved through the following technical solutions:

[0014] A multi-camera collaborative calibration method suitable for dynamic vision sensors, characterized in that it includes the following steps:

[0015] S1, obtain sub-pixel corner point set information, according to the frame image taken by the frame camera and the image reconstructed by the event, send it to the neural network, and obtain the sub-pixel corner point set information,

[0016] S2, optimize the camera parameters to minimize the reprojection error and thus improve the accuracy of the camera model: calculate the fitness function to measure the reprojection error between the calibrated image points and the actual world coordinate points, use the evolutionary algorithm to iteratively solve through crossover and mutation operations, retain the parameter configuration with high fitness, and gradually optimize these parameters to minimize the reprojection error, calculate the camera's intrinsic parameter matrix and distortion parameters, and use them to remove camera distortion;

[0017] S3, according to the sub-pixel corner point set information, a random sampling consensus algorithm is used to calculate a first transformation matrix and a second transformation matrix, wherein the first matrix is ​​a transformation matrix of the frame camera relative to the origin of the reference coordinate system, and the second matrix is ​​a transformation matrix of the event camera relative to the origin of the reference coordinate system;

[0018] S4, solving the transformation matrix from the event camera pixel space to the frame camera pixel space according to the obtained first transformation matrix and the second transformation matrix.

[0019] While adopting the above technical solutions, the present invention may also adopt or combine the following technical solutions:

[0020] As a preferred technical solution of the present invention: in step S1, obtaining sub-pixel corner point set information includes the following steps:

[0021] S1.1, using a time-synchronized frame camera and event camera to shoot the calibration plate under the illumination of a light source flickering at a constant frequency, to obtain a static calibration image and dynamic event stream information;

[0022] S1.2, performing correlation noise filtering and time surface integral on the dynamic event flow information to obtain a grayscale image;

[0023] S1.3, performing corner point information detection on the static calibration image and the grayscale image.

[0024] As a preferred technical solution of the present invention: in step S1.2, correlation noise filtering is performed on the dynamic event stream information, including:

[0025] In a tiny neighborhood, select events to form an event set, and calculate the similarity of events in the event set based on the location information, timestamp information, and polarity information of the events in the set;

[0026] The events are subjected to noise judgment according to the similarity of the events in the event set, and the noise events are eliminated.

[0027] As a preferred technical solution of the present invention: in step S1.2, the event set information after noise removal is time-integrated to obtain a reconstructed integral image;

[0028] The reconstructed image is obtained by integrating the filtered event timestamp, event pixel coordinates and event polarity information.

[0029] As a preferred technical solution of the present invention: in step S1.3, the corner point detection method includes:

[0030] Preprocess the input frame image and the reconstructed integral image and convert them into single-channel grayscale images;

[0031] The pre-trained ResNet-50 model is used to extract image features and generate a heat map representing the probability of corner point locations, where each channel corresponds to a corner point on the chessboard;

[0032] The heat map output is fed into a customized multi-layer convolutional network to strengthen the local feature representation of corner points;

[0033] Through network optimization, the prominence of corner points on the heat map is enhanced, making the positions of corner points more prominent;

[0034] Finally, the corner points are located by applying global maximum pooling along the x and y dimensions on the processed heat map to obtain a set of corner point information.

[0035] As a preferred technical solution of the present invention: in step S2, the reprojection error detection includes the following steps:

[0036] Define the reprojection error function, that is, the fitness function is the detected corner point position p i and the corner point position p′ after projection by the projection model M i The two-dimensional Euclidean distance between:

[0037] Calculating the fitness function to measure the reprojection error between the calibrated image points and the actual world coordinate points includes the following steps:

[0038] S2.1, for each camera calibration parameter configuration, calculate the reprojection error of the corner points in all calibration images;

[0039] S2.2, accumulating or averaging all reprojection errors to obtain a total reprojection error for the camera calibration parameter configuration;

[0040] S2.3, through the crossover and mutation operations in the genetic algorithm, the parameter configurations with high fitness are retained, and these parameters are gradually optimized to minimize the reprojection error;

[0041] S2.4, through the iterative optimization process of the genetic algorithm, the camera's intrinsic parameter matrix and distortion parameters are calculated to remove the camera distortion, thereby ensuring that the image data collected by the camera is accurate and reliable.

[0042] As a preferred technical solution of the present invention: in step S3, the joint calibration of the frame camera and the event camera requires the internal parameters and distortion matrix calibration of the two cameras in advance.

[0043] Firstly, an external clock is used to connect the event camera and the frame camera to synchronize the time of the two cameras. The synchronized event camera and frame camera are used to shoot the calibration plate under the illumination of a flashing light source to obtain a static calibration image Img i and dynamic event stream information e respectively. After the static calibration image is dedistorted, corner point detection is performed, and the camera posture relative to the world coordinate system, i.e., the first transformation matrix R1 and the second transformation matrix R2, are solved according to the corner point set. The parameters of the first transformation matrix and the second transformation matrix are iteratively solved by a random sampling method to improve the accuracy of the calculation results. The transformation matrix from the event camera to the frame camera pixel space is obtained by using the first transformation matrix and the second transformation matrix.

[0044] As a preferred technical solution of the present invention: in step S3, the first transformation matrix and the second transformation matrix of the camera are solved:

[0045] S3.1 Calculate the world coordinate origin: Calculate the world coordinate origin information based on the detected corner point information set;

[0046] S3.2 Select calibration points: select the corner points after the distortion correction as the corner point set, and randomly select a certain number of point coordinates from the set as calibration points;

[0047] S3.3 Coordinate normalization: For each calibration point, the pixel coordinates (u, v) in the image are converted into normalized plane coordinates (x, y) using the camera intrinsic parameters. Where fx, fy are the focal lengths of the camera, cx, cy are the coordinates of the principal point;

[0048] S3.4 Solve the first / second transformation matrix: Use the world coordinate points and the normalized plane coordinates, use the nonlinear least squares method and the singular value decomposition method to calculate the first / second transformation matrix;

[0049] S3.5 For the remaining data points, check whether they match the model and calculate the number of data points that match the model for evaluating the model;

[0050] S3.6 Repeat steps 3.2-3.5 multiple times and select the model parameters with the best evaluation effect as the final result.

[0051] As a preferred technical solution of the present invention: in step S4, the transformation matrix of the event camera mapped to the frame camera space is solved:

[0052] The intrinsic parameter matrix K1 of the frame camera, the intrinsic parameter matrix K2 of the event camera, the sum distortion matrix D1 of the frame camera and the sum distortion matrix D2 of the event camera, as well as the first transformation matrix R1 and the second transformation matrix R2, where u and v are the pixel coordinates of the event camera data in the frame camera image, that is, the converted coordinates, and the transformation matrix T is: T = K1*D1 -1 R2*R1*D2 -1 *K2, then fine-tune the transformation matrix T and select the gradient descent method to optimize the internal parameters of the matrix.

[0053]

[0054] The second object of the present invention is to provide an application of a multi-camera collaborative calibration method suitable for dynamic vision sensors in view of the problems in the prior art.

[0055] To this end, the above-mentioned purpose of the present invention is achieved through the following technical solutions:

[0056] The application of the multi-camera collaborative calibration method suitable for dynamic visual sensors is characterized by: being applied to the calibration of multi-camera systems for virtual reality, augmented reality and mixed reality, being applied to synchronous positioning and perspective generation in three-dimensional space, generating new perspectives in three-dimensional space, or being applied to eye tracking in modeling the motion posture of the human eye in 3D space.

[0057] The present invention relates to camera calibration technology, which is applicable to the fields of three-dimensional reconstruction and measurement, augmented reality (AR) and virtual reality (VR), camera correction and image processing.

[0058] Compared with the prior art, the present invention provides a multi-camera collaborative calibration method and application suitable for dynamic vision sensors, and proposes a multi-camera efficient collaborative calibration method suitable for dynamic vision sensors. The corner detection method of the present invention can achieve sub-pixel corner detection accuracy, provide reliable results for camera internal and external parameters, and more accurately capture camera parameters and scene information, thereby providing a more reliable basis for subsequent visual tasks. This high-precision calibration method is of great significance in various application fields, and can be applied to spatial calibration between frame cameras and event cameras, and can be applied to calibration between 1-multiple event cameras and 1-multiple frame cameras, and cameras at different angles and positions, so that the multi-camera system can perform multi-perspective sensor calibration in three-dimensional space, thereby improving the overall data fusion accuracy and robustness of the multi-camera system, and has great application prospects in the calibration of multi-camera systems in virtual reality, augmented reality and mixed reality, synchronous positioning and perspective generation in three-dimensional space, generation of new perspectives in three-dimensional space, and eye tracking of human eye motion posture modeling in 3D space. Specifically,

[0059] 1. Obtain initial pixel-level corner information through frame images taken by the frame camera and images reconstructed based on event camera data, and locate the corners accurately to the sub-pixel level through a neural network, thus providing a more reliable foundation for subsequent visual tasks;

[0060] 2. A method combining a neural network and a genetic algorithm, wherein the neural network is used to extract and optimize the position information of sub-pixel corner points from two image sources, providing accurate initial data for the genetic algorithm; the genetic algorithm performs a global search, and through multiple rounds of iterations, the genetic algorithm gradually eliminates parameter configurations with low fitness, retains and optimizes configurations with high fitness, so as to find the optimal camera parameters; the present invention makes full use of the powerful ability of deep learning in feature extraction and the efficiency of genetic algorithms in global optimization search, thereby achieving high-precision camera calibration and image correction, and minimizing the reprojection error by adjusting and optimizing the camera's intrinsic parameter matrix and distortion parameters to ensure the accuracy and reliability of the image data. After the camera group's intrinsic parameters and distortion parameters are determined, they are used to remove or correct the distortion in the image, and the processed image can more accurately reflect the actual scene.

[0061] 2. When calibrating the camera through corner points, the random sampling consensus method can effectively screen out the correct corner points, ignore the wrong detection, improve the calibration speed, and accurately fit the camera internal parameters and distortion parameters directly for image correction, eliminate distortion, improve the accuracy of subsequent image processing steps, reduce the amount of data and ensure the representativeness of the sample, thereby improving the computational efficiency. At the same time, through the iterative process, the model can be continuously optimized to improve the accuracy of the pose matrix. The combination uses the gradient descent method to fine-tune the internal parameters of the transformation matrix, and through continuous iteration to find the optimal parameters, the objective function is minimized, further optimizing the accuracy of the transformation matrix, making the final result more reliable.

[0062] 3. The calibration method of the present application does not rely on complex calibration images. Compared with traditional calibration methods, it does not require a specific calibration plate or a special calibration environment, which reduces the limitations and difficulties that may be encountered in practical applications and makes the calibration process smoother. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is a single camera calibration-frame camera calibration method;

[0064] Figure 2 A method for single camera calibration and event camera calibration;

[0065] Figure 3 A schematic diagram of a multi-camera collaborative calibration method for a dynamic vision sensor of the present invention;

[0066] Figure 4 Optical path diagram for collaborative calibration. DETAILED DESCRIPTION

[0067] The present invention will be described in further detail with reference to the accompanying drawings and specific embodiments.

[0068] First, an external clock is used to connect the event camera and the frame camera, and the two cameras are synchronized to ensure that the timestamp of the event stream and the timestamp of the frame data are aligned. The synchronized event camera and frame camera are used to shoot the calibration plate under the illumination of the flashing light source to obtain the static calibration image Img i and the dynamic event stream information e respectively. By changing the external lighting environment instead of generating events through relative movement between the camera and the calibration plate, this method can not only generate rich event data, but also effectively reduce the errors caused by motion and obtain more accurate event information.

[0069] Then, correlation noise filtering is performed on the dynamic event stream information to reduce the impact of noise on corner point detection. The similarity between events is calculated based on the pixel position information, timestamp and polarity information of the event, that is, the event similarity is calculated based on the time interval and spatial interval between events, that is, the distance between pixels, to determine whether the event is noise. When there are less than 8 similar events near a small neighborhood in the time window, the event can be considered as noise and removed.

[0070] After the correlation filtering is completed, the time integral is constructed based on the time integral method according to the timestamp of the event. When constructing the time integral, it is necessary to set the integration time T and the decay function f(x, y, τ). The integration time T is generally set to 33ms. The decay function f(x, y, τ) can be a step function, an exponential function, or a linear function. The decay function contains the horizontal and vertical coordinates of the pixel and the decay constant. For example, when the integration time T = 33ms and the exponential decay function is selected, the time surface Among them I t (x, y) is the brightness of the pixel at the position (x, y) at time t. The result of the time surface integral is output as an image containing corner point information.

[0071] The static images taken by the frame camera and the images obtained by time integration of event data are preprocessed and converted into grayscale images; the grayscale images are then input into the pre-trained ResNet model to extract image features, and the output layer is modified to generate a heat map representing the probability of corner position. These heat maps further enhance the local feature representation of corners through a customized multi-layer convolutional network. The corner positions with the highest probability are extracted along the x and y dimensions on the heat map through global maximum pooling to generate a set of corner information. The mean square error MSE is used as the loss function in the training phase, and the Adam optimizer is used to optimize the network parameters to reduce the deviation between the predicted coordinates and the true annotations. This neural network method can effectively detect the corner information of the checkerboard and improve the detection accuracy.

[0072] Subsequently, the fitness function is calculated based on the obtained sub-pixel corner point set information to measure the reprojection error between the calibrated image points and the actual world coordinate points. Through crossover and mutation operations, the evolutionary algorithm is used to iteratively solve, retain the parameter configuration with high fitness, and gradually optimize these parameters to minimize the reprojection error. The intrinsic parameter matrix and distortion parameters of the camera are calculated to remove camera distortion. For each camera calibration parameter configuration, the reprojection error of the corner points in all calibrated images is calculated, and all reprojection errors are accumulated or averaged to obtain the total reprojection error of the camera calibration parameter configuration. Through the crossover and mutation operations in the genetic algorithm, the parameter configuration with high fitness is retained, and these parameters are gradually optimized to minimize the reprojection error. Through the iterative optimization process of the genetic algorithm, the intrinsic parameter matrix and distortion parameters of the camera can be calculated, which can be used to remove camera distortion, thereby ensuring that the image data collected by the camera is more accurate and reliable.

[0073] The camera's posture relative to the world coordinate system is solved based on the set of corner points, namely the first transformation matrix R1 and the second transformation matrix R2. In order to improve the accuracy of the calculation results, the random sampling method is used to iteratively solve the first / second transformation matrix parameters. First, a certain number of point coordinates are randomly selected from the corner point information set as calibration points, and the world coordinate points and normalized plane coordinates are used to solve the first / second transformation matrix using the nonlinear least squares method and the singular value decomposition method. For the remaining data points, check whether they match the model, calculate the number of data points that meet the model, and use them to evaluate the model; repeat the above steps multiple times, and select the model parameters with the best evaluation effect as the final result.

[0074] Finally, the transformation matrix T from the event camera to the frame camera pixel space is obtained using the first transformation matrix and the second transformation matrix. The gradient descent method is used to optimize the internal parameters of the matrix and fine-tune the transformation matrix.

[0075] Example 1

[0076] like Figure 1-Figure 4 As shown, in a multi-camera collaborative calibration method applicable to a dynamic vision sensor of the present invention:

[0077] Definition of frame camera and event camera:

[0078] A frame camera, also called a still camera or full-frame camera, is a camera that captures the entire image scene each time the shutter is triggered, with the output being an RGB three-channel or grayscale single-channel image.

[0079] The event camera is a new type of bio-inspired visual sensor, sometimes also called a dynamic visual sensor DVS or DAVIS. It works completely differently from traditional cameras in terms of data return format. Instead of outputting intensity image frames at a constant rate, the event camera only outputs information related to local pixel-level brightness changes. The event camera will only generate output when the brightness change of the pixel is greater than the set threshold. Its output is event stream information, which contains 4 types of information: e = {x, y, t, p}, that is, the horizontal and vertical coordinates of the event (x, y), the timestamp information of the event, and the polarity information of the event. The polarity information includes two types {+1, -1}. When the brightness increment of the pixel is greater than the threshold, the polarity of the event is +1 polarity, and when the brightness decrement of the pixel is greater than the threshold, the polarity of the event is -1 polarity.

[0080] Single Camera Calibration Method

[0081] For the camera calibration of the frame camera, first set the optical parameters of the camera, including exposure time, camera focal length, frame rate and other parameters. Then use the frame camera to collect n static chessboard images I i , generally 15-20 images are taken. Corner point information detection is performed on static frame images. Its input is three-channel information or single-channel image information, and its output is the image corner point information p i Based on the corner point information, a projection model from the three-dimensional world to the two-dimensional world is established. The parameters of the projection model include the camera intrinsic parameter information, distortion information and camera extrinsic parameters. The camera intrinsic parameter information includes the focal length and the position of the principal point; the distortion information includes radial distortion and tangential distortion; the camera extrinsic parameters include the rotation and translation matrix of the camera relative to the world coordinate system. The reprojection error function is defined as the detected corner point position p i and the corner point position p′ after projection by the projection model M i The two-dimensional Euclidean distance between: By minimizing the size of the reprojection error function, the parameters of the projection model M are solved. Corner point information detection method:

[0082] The pre-trained ResNet-50 model is used to extract image features. The input is the frame image and the output is the coordinate value of the corner point information.

[0083] In order to adapt to the specific needs of checkerboard corner detection, the architecture of ResNet was modified, mainly by replacing the original output layer. The output layer originally used for classification was modified to generate heat maps representing the probability of corner position. Each channel in these heat maps corresponds to a corner point on the checkerboard. This adjustment overcomes the accuracy limitations of traditional methods in corner information detection and provides an intuitive representation of the potential position of corner information.

[0084] These heat map outputs are fed into a customized multi-layer convolutional network to further enhance the local feature representation of corner points. The specific steps are as follows:

[0085] Through network optimization, the saliency of corner points on the heat map is enhanced, making the location of corner points more prominent. This step enhances the local differentiation of corner points, allowing the model to locate corner points more accurately in complex backgrounds.

[0086] Finally, the localization of the corner points is achieved by applying global maximum pooling along the x and y dimensions on the processed heat map, ensuring that the coordinates extracted from each dimension are the corner points with the highest probability. This optimization ensures the accurate extraction of the corner point positions and reduces the spatial bias of the model in corner point identification.

[0087] In the training phase of the model, the mean square error (MSE) is used as the loss function, and the Adam optimizer is used to iteratively optimize the network parameters to reduce the deviation between the predicted coordinates and the true annotations. This strategy reduces the error between the predicted coordinates and the true annotations, and improves the overall accuracy and generalization ability of the model.

[0088] This method uses a neural network to detect checkerboard corner information from frame images. This step is based on the feature extraction capability of deep learning, capturing the complex texture information in the image and laying the foundation for accurate corner detection. The combined application of these steps not only improves the front-end accuracy of corner detection, but also provides more reliable input data for subsequent image processing processes, greatly enhancing the performance and efficiency of the overall system.

[0089] Imaging projection model establishment:

[0090] According to the corner detection results, an imaging model is selected. The available imaging projection models include a pinhole model and an omnidirectional model.

[0091] Taking the pinhole model as an example, the pinhole model involves four parameters: the horizontal focal length fx, the vertical focal length fy, also known as the imaging zoom parameter; the horizontal principal point coordinate cx, the vertical principal point coordinate cy, also known as the imaging offset parameter. These four parameters together constitute the camera's intrinsic parameter matrix K.

[0092] A point P (X, Y, Z) in the camera coordinate system o-xyz space passes through the optical center and obtains an imaging point on the physical imaging plane. The projection point p (u, v) on the pixel plane is finally obtained by scaling and offsetting, which can be expressed in matrix language as follows:

[0093]

[0094] After establishing the imaging projection model, the camera needs to be distorted. Generally, the distortion of the camera lens includes tangential distortion and radial distortion.

[0095] The radial distortion is related to the shape of the lens, and the severity of the distortion is related to the distance from the optical center. It is mainly divided into barrel distortion and pincushion distortion. Three distortion parameters k1, k2, and k3 are introduced to correct the tangential distortion. First, the coordinates of a point (x, y) on the normalized plane are (r, θ) in the polar coordinate system. The distortion is described by quadratic and high-order polynomials. The parameter k1 plays a major role in the center area, and the parameter k2 plays a major role in the edge area with large distortion. For cameras with large distortion, k3 can also be added for distortion correction:

[0096]

[0097] Tangential distortion is caused by the non-strict parallelism between the lens and the imaging plane, and is corrected using two other parameters p1 and p2:

[0098]

[0099] After considering the tangential and radial distortions, the coordinates of the points on the original image can be obtained:

[0100]

[0101] Finally, the intrinsic parameter is projected into the pixel space to obtain the position of the point on the image:

[0102]

[0103] Reprojection error detection

[0104] Define the reprojection error function, that is, the fitness function, as the detected corner point position p i and the corner point position p′ after projection by the projection model M i The two-dimensional Euclidean distance between:

[0105] To calculate the fitness function, which measures the reprojection error between the calibrated image points and the actual world coordinates, use the following steps:

[0106] (1) For each camera calibration parameter configuration, calculate the reprojection error of the corner points in all calibration images;

[0107] (2) Accumulate or average all reprojection errors to obtain the total reprojection error of the camera calibration parameter configuration;

[0108] (3) Through the crossover and mutation operations in the genetic algorithm, the parameter configurations with high fitness are retained and these parameters are gradually optimized to minimize the reprojection error;

[0109] (4) Through the iterative optimization process of the genetic algorithm, the camera's intrinsic parameter matrix and distortion parameters can be calculated. These parameters can be used to remove camera distortion, thereby ensuring that the image data collected by the camera is more accurate and reliable.

[0110] Event Camera Calibration Method

[0111] For the camera calibration of the event camera, the optical parameters of the camera are first set, including exposure time, camera focal length, and sensor sensitivity. Since the event camera does not output intensity image frames at a constant rate, but only outputs information related to local pixel-level brightness changes, the event camera will only generate output when the brightness change of the pixel is greater than the set threshold, and the output is recorded as an event. The sensor sensitivity mentioned above is the set brightness threshold.

[0112] When using an event camera to shoot the calibration checkerboard, an additional triggering light source is required. The light source faces the checkerboard (the light source can include near-infrared and visible light bands) and provides illumination light at a fixed flickering frequency, so that the event camera can receive the brightness changes caused by flickering. Use the event camera to continuously shoot the calibration checkerboard to generate dynamic event stream information, then perform correlation filtering on the dynamic event stream information, and then perform event surface integral on the filtered events to generate a single-channel grayscale image. Then, the single-channel grayscale image is subjected to corner point information detection. Then, a projection model from the three-dimensional world to the two-dimensional world is established based on the corner point information. The parameters of the projection model include camera intrinsic parameter information and distortion information. The projection model parameters are solved by minimizing the reprojection error function. This step is the same as frame camera calibration.

[0113] Correlation Noise Filtering

[0114] Perform correlation noise filtering on dynamic event stream information to reduce the impact of noise on corner point detection. Calculate the similarity between events based on the pixel position information, timestamp and polarity information of the event, that is, divide the event set according to the time interval and spatial interval between events, that is, the distance between pixels, and then calculate the similarity of events in the event set to determine whether the event is noise. When there are less than 8 similar events near a small neighborhood in the time window, the event can be considered as noise and removed.

[0115] After the correlation filtering is completed, the time integral is constructed according to the time stamp of the event using a time integral-based method.

[0116] When constructing a time integral, you need to set the integration time T and the decay function f(x, y, τ). The integration time T is generally set to 33ms. The decay function f(x, y, τ) can be a step function, exponential function, or linear function. The decay function contains the horizontal and vertical coordinates of the pixel and the decay constant. For example, when the integration time T = 33ms and the exponential decay function is selected, the time surface Among them I t (x, y) is the brightness of the pixel at the position (x, y) at time t.

[0117] The result of the time surface integral is output as an image containing corner point information.

[0118] The joint calibration of the frame camera and the event camera requires the calibration of the internal parameters and distortion matrices of the two cameras in advance. First, an external clock is used to connect the event camera and the frame camera to synchronize the time of the two cameras. Use the synchronized event camera and frame camera to shoot the calibration plate under the illumination of the flashing light source, and obtain the static calibration image Img i and the dynamic event stream information e respectively. After the static calibration image is dedistorted, corner point detection is performed, and the camera posture relative to the world coordinate system, that is, the first transformation matrix R1 and the second transformation matrix R2, are solved according to the corner point set. In order to improve the accuracy of the calculation results, the random sampling method is used to iteratively solve the first / second transformation matrix parameters. The transformation matrix from the event camera to the frame camera pixel space is obtained using the first transformation matrix and the second transformation matrix.

[0119] Through high-precision transformation matrix calculation, not only can the quality of data fusion be significantly improved, but also a unified and accurate spatial context can be provided for information from multiple data sources. This precise data fusion technology enables data obtained from different cameras to be accurately aligned and fused, greatly improving the accuracy and realism of 3D reconstruction. In 3D reconstruction, this means being able to generate more detailed and accurate models, which is crucial for virtual reality applications in cultural heritage protection, the construction industry, and the entertainment industry.

[0120] In augmented reality (AR) applications, accurate spatial positioning and angle estimation not only improve the integration of virtual content with the real world, but also greatly enhance the naturalness and intuitiveness of user interaction. For example, the fields of education and training can use this technology to improve the efficiency and effectiveness of learning by virtually simulating actual operation scenarios. In the field of robot navigation and autonomous driving, accurate transformation matrix calculation allows the system to understand its surrounding environment more accurately, thereby performing safer and more effective path planning. For autonomous vehicles, the application of this technology can reduce the misunderstanding of sensor inputs and reduce the risk of potential accidents, while optimizing driving routes and traffic management systems. In addition, accurate transformation matrices can also help better implement intelligent monitoring and security systems, improving the effectiveness and response speed of monitoring systems by more accurately tracking and identifying objects. In disaster response and rescue operations, this technology can provide real-time, accurate environmental mapping to guide rescue teams, significantly improving rescue efficiency and safety. Therefore, this high-precision transformation matrix calculation is not only a technological advancement, but also brings a wide range of possibilities and significant advantages to practical applications in various industries.

[0121] The camera's first transformation matrix and second transformation matrix are solved:

[0122] 1. Calculate the world coordinate origin: Calculate the world coordinate origin information based on the detected corner point information set;

[0123] 2. Select calibration points: select the corner points after distortion correction as the corner point set, and randomly select a certain number of point coordinates from the set as calibration points;

[0124] 3. Coordinate normalization: For each calibration point, the pixel coordinates (u, v) in the image are converted into normalized plane coordinates (x, y) using the camera intrinsic parameters. Where fx, fy are the focal lengths of the camera, cx, cy are the coordinates of the principal point;

[0125] 4. Solve the first / second transformation matrix: Use the world coordinate points and normalized plane coordinates, use the nonlinear least squares method and the singular value decomposition method to calculate the first / second transformation matrix.

[0126] 5. For the remaining data points, check whether they match the model and calculate the number of data points that match the model for evaluating the model;

[0127] 6. Repeat steps 2-5 multiple times and select the model parameters with the best evaluation effect as the final result.

[0128] Solve the transformation matrix that maps the event camera to the frame camera space:

[0129] To solve the transformation matrix, the information needed includes: the intrinsic parameter matrix and distortion matrix K1, D1, K2, D2 of the frame camera and event camera, as well as the first transformation matrix and the second transformation matrix R1, R2, where u and v are the pixel coordinates of the event camera data in the frame camera image, that is, the coordinates after conversion, and the transformation matrix T is: T = K1*D1 -1 R2*R1*D2 -1 *K2, then fine-tune the transformation matrix T and select the gradient descent method to optimize the internal parameters of the matrix. The internal parameters of the transformation matrix are optimized by the gradient descent method to ensure that the optimal effect can be achieved in practical applications. This step not only improves the stability of the model, but also ensures its adaptability in dynamic environments.

[0130]

[0131] The above-mentioned specific implementation methods are used to explain the present invention and are only preferred embodiments of the present invention, rather than limiting the present invention. Any modifications, equivalent substitutions, improvements, etc. made to the present invention within the spirit of the present invention and the protection scope of the claims shall fall within the protection scope of the present invention.

Claims

1. A multi-camera collaborative calibration method for dynamic vision sensors. Features: The following steps are involved: S1, obtaining sub-pixel corner point set information: the frame image taken by the frame camera and the event-reconstructed image are transmitted to the neural network to obtain the sub-pixel corner point set information; S2, optimize the camera parameters to minimize the reprojection error and thus improve the accuracy of the camera model: calculate the fitness function to measure the reprojection error between the calibrated image points and the actual world coordinate points, use the evolutionary algorithm to iteratively solve through crossover and mutation operations, retain the parameter configuration with high fitness, and gradually optimize these parameters to minimize the reprojection error, calculate the camera's intrinsic parameter matrix and distortion parameters, and use them to remove camera distortion; S3, according to the sub-pixel corner point set information, a random sampling consensus algorithm is used to calculate a first transformation matrix and a second transformation matrix, wherein the first matrix is ​​a transformation matrix of the frame camera relative to the origin of the reference coordinate system, and the second matrix is ​​a transformation matrix of the event camera relative to the origin of the reference coordinate system; S4, solving a transformation matrix from an event camera pixel space to a frame camera pixel space according to the obtained first transformation matrix and the second transformation matrix; In step S4, the transformation matrix of the event camera mapped to the frame camera space is solved: The intrinsic parameter matrix K1 of the frame camera, the intrinsic parameter matrix K2 of the event camera, the distortion matrix D1 of the frame camera and the distortion matrix D2 of the event camera, as well as the first transformation matrix R1 and the second transformation matrix R2, where u and v are the pixel coordinates of the event camera data in the frame camera image, that is, the converted coordinates, and the transformation matrix T is: T = K1*D1 -1 R1*R2*D2 -1 *K2, then fine-tune the transformation matrix T and select the gradient descent method to optimize the internal parameters of the matrix. Among them, (X, Y, Z) are the coordinates of the data in the event camera coordinate system o-xyz space.

2. The multi-camera collaborative calibration method for dynamic vision sensors according to claim 1, Features: In step S1, obtaining sub-pixel corner point set information includes the following steps: S1.1, using a time-synchronized frame camera and event camera to shoot the calibration plate under the illumination of a light source flickering at a constant frequency, to obtain a static calibration image and dynamic event stream information; S1.2, performing correlation noise filtering on the dynamic event stream information to obtain a noise-removed event set; and performing time surface integral to obtain a grayscale image; S1.3, performing corner point information detection on the static calibration image and the grayscale image.

3. The multi-camera collaborative calibration method for dynamic vision sensors according to claim 2, Features: In step S1.2, correlation noise filtering is performed on the dynamic event stream information, including: In a tiny neighborhood, select events to form an event set, and calculate the similarity of events in the event set based on the location information, timestamp information, and polarity information of the events in the set; The events are subjected to noise judgment according to the similarity of the events in the event set, and the noise events are eliminated to obtain the event set after the noise is eliminated.

4. The multi-camera collaborative calibration method for dynamic vision sensors according to claim 2, Features: In step S1.2, a time surface integral is performed to obtain a grayscale image, and a time integral is performed on the events in the event set after noise removal according to the polarity and position information of the events to obtain a reconstructed integral image: the events in the event set after noise removal are accumulated in chronological order to form a time surface, the neighborhood density of local events is calculated, and the event stream is converted into a grayscale image through a surface integral algorithm.

5. The multi-camera collaborative calibration method for dynamic vision sensors according to claim 2, Features: In step S1.3, the corner point detection method includes: Preprocess the input frame image and the reconstructed integral image and convert them into single-channel grayscale images; The pre-trained ResNet-50 model is used to extract image features and generate a heat map representing the probability of corner point locations, where each channel corresponds to a corner point on the chessboard; The heat map output is fed into a customized multi-layer convolutional network to strengthen the local feature representation of corner points; Through network optimization, the prominence of corner points on the heat map is enhanced, making the positions of corner points more prominent; Finally, the corner points are located by applying global maximum pooling along the x and y dimensions on the processed heat map to obtain a set of corner point information.

6. The multi-camera collaborative calibration method applicable to dynamic vision sensors according to claim 1, Features: In step S2, the reprojection error detection includes the following steps: Define the reprojection error function, that is, the fitness function is the detected corner point position p i and the corner point position p′ after projection by the projection model M i The two-dimensional Euclidean distance between: Calculating the fitness function to measure the reprojection error between the calibrated image points and the actual world coordinate points includes the following steps: S2.1, for each camera calibration parameter configuration, calculate the reprojection error of the corner points in all calibration images; S2.2, accumulate or average all the reprojection errors to obtain the total reprojection error of the camera calibration parameter configuration; S2.3, through the crossover and mutation operations in the genetic algorithm, the parameter configurations with high fitness are retained, and these parameters are gradually optimized to minimize the reprojection error; S2.4, through the iterative optimization process of the genetic algorithm, the camera's intrinsic parameter matrix and distortion parameters are calculated to remove the camera distortion, thereby ensuring that the image data collected by the camera is accurate and reliable.

7. The multi-camera collaborative calibration method for dynamic vision sensors according to claim 1, Features: In step S3, the joint calibration of the frame camera and the event camera requires the calibration of the intrinsic parameters and distortion matrix of the two cameras in advance; Firstly, an external clock is used to connect the event camera and the frame camera to synchronize the time of the two cameras. The synchronized event camera and frame camera are used to shoot the calibration plate under the illumination of a flashing light source to obtain a static calibration image Img i and dynamic event stream information e respectively. After the static calibration image is dedistorted, corner point detection is performed, and the camera posture relative to the world coordinate system, i.e., the first transformation matrix R1 and the second transformation matrix R2, are solved according to the corner point set. The parameters of the first transformation matrix and the second transformation matrix are iteratively solved by a random sampling method to improve the accuracy of the calculation results. The transformation matrix from the event camera to the frame camera pixel space is obtained by using the first transformation matrix and the second transformation matrix.

8. The multi-camera collaborative calibration method for dynamic vision sensors according to claim 7, Features: In step S3, the first transformation matrix and the second transformation matrix of the camera are solved: S3.1 Calculate the world coordinate origin: Calculate the world coordinate origin information based on the detected corner point information set; S 3.2 Select calibration points: select the corner points after the distortion correction as the corner point set, and randomly select a certain number of point coordinates from the set as calibration points; S 3.3 Coordinate normalization: For each calibration point, the pixel coordinates (u, v) in the image are converted into normalized plane coordinates (x, y) using the camera intrinsic parameters. Where fx, fy are the focal lengths of the camera, cx, cy are the coordinates of the principal point; S 3.4 Solve the first / second transformation matrix: Use the world coordinate points and the normalized plane coordinates, use the nonlinear least squares method and the singular value decomposition method to calculate the first / second transformation matrix; S 3.5 For the remaining data points, check whether they match the model and calculate the number of data points that match the model for evaluating the model; S 3.6 Repeat steps 3.2-3.5 multiple times and select the model parameters with the best evaluation effect as the final result.

Images