Spatial alignment method for a combination camera based on a traditional camera and an event camera

Abstract

The application relates to a space alignment method based on a combination camera of a traditional camera and an event camera, and belongs to the technical field of cameras. When event stream data generated by the event camera is processed, the space alignment method generates a deblurred event frame by projecting and compensating events, so that the definition of the event frame image and the scene detail representation capability are improved. The frame image of the traditional camera is down-sampled and an edge gradient image is calculated. Finally, the event image and the frame image are normalized and the sliding matching calculation of cross correlation is performed, so that a sub-pixel level precision disparity map is obtained, and high-precision space alignment of the traditional camera and the event camera is realized.

Description

Technical Field

[0001] The present invention relates to the technical field of camera devices, and in particular to a spatial alignment method of a camera device based on a combination of a traditional camera and an event camera. Background Art

[0002] Spatial alignment of traditional cameras and event cameras is a prerequisite for effective multimodal data fusion in heterogeneous vision systems. Its core purpose is to eliminate spatial observation inconsistencies caused by imaging conditions and spatial layouts between two sensors using different principles. Spatial alignment of two cameras ensures precise correspondence between asynchronous event streams and frame images generated in the same scene within the same coordinate system, thereby establishing geometric consistency between the data from the two cameras. The accuracy of this spatial alignment directly impacts the accuracy of heterogeneous vision systems in tasks such as motion estimation and 3D reconstruction.

[0003] Spatial alignment between traditional cameras and event cameras requires overcoming differences in data modality and spatiotemporal resolution between the two sensors. Event cameras asynchronously output event streams, which reflect changes in pixel brightness. Event stream data is sparse and based on edge motion. Directly matching this with the frame images output by traditional cameras can lead to matching ambiguity due to sparsity. Therefore, achieving high-quality processing of event stream data is a critical issue.

[0004] Existing spatial alignment methods for traditional cameras and event cameras primarily rely on binocular matching or spectrometer systems. The former calculates disparity through feature point detection (such as ORB and SIFT) or structural similarity (SSIM) matching, but this approach is computationally complex. While the latter allows the two cameras to share an optical path, the physical positions of the two sensors cannot completely overlap. Manufacturing and installation errors in the spectrometers also result in slight spatial offsets between the two cameras' imaging sensors. In heterogeneous vision systems combining traditional and event cameras, high-precision spatial alignment that aligns the data characteristics of both cameras and leverages their complementary strengths remains a challenge. Summary of the Invention

[0005] The present invention addresses the shortcomings of spatial alignment between two cameras in current imaging systems and proposes a spatial alignment method based on a combined imaging device of a traditional camera and an event camera. This method processes the event stream data generated by the event camera by projecting and compensating events to generate deblurred event frames, improving the clarity of the event frame images and their ability to represent scene details. The frame images from the traditional camera are then downsampled and edge gradient images are calculated. Finally, a normalized cross-correlation sliding matching calculation is performed on the event image and frame image to obtain a sub-pixel disparity map, achieving high-precision spatial alignment of the traditional and event cameras.

[0006] The present invention provides a spatial alignment method based on a combined imaging device of a traditional camera and an event camera, comprising the following steps:

[0007] Step 101: spatially aligning a conventional camera and an event camera to obtain an event stream captured by the time-synchronized event camera and corrected by trigger timestamps, and an original frame image captured by the conventional camera and corrected by time alignment and trigger timestamps.

[0008] Step 201: Process the event stream to generate a motion-blurred event frame;

[0009] Step 301, processing the original frame image to generate a downsampled edge gradient image;

[0010] Step 401: De-motion blurring of event frames based on event cameras G and edge gradient images from traditional cameras F The similarity matching calculation obtains sub-pixel disparity;

[0011] Step 501 : Calculate sub-pixel disparity pixel by pixel and generate an optimized sub-pixel disparity map.

[0012] Optionally, the specific steps of step 201 are as follows:

[0013] Step 2011. Determine an initial timestamp tag table and an initial event image with the same resolution as the event camera;

[0014] Step 2012. Sequentially a events Projecting onto the event image and performing motion compensation on the event image to obtain a compensated event image ,in, Respectively represent a Event x Axis coordinates, y axis coordinates and timestamps;

[0015] Step 2013. Compensate event image Perform normalization to generate motion-blurred event frames G .

[0016] Optionally, the specific steps of step 2012 are as follows:

[0017] Step 20121. Order a =0, when a =0, it is the initial event, the corresponding current timestamp mark table is the initial timestamp mark table, and the current event image is the initial event image;

[0018] Step 20122. Get the first a events The current timestamp marks the coordinates of the table The coordinate value of ;

[0019] Step 20123. a Project the event to the current event image and obtain the projected event image ;

[0020] Step 20124. a Perform motion compensation on each event to obtain the current compensated event image ;

[0021] Step 20125. Judgment a With the size of the total number of events A, if a Greater than or equal to A, then use the current compensation event image ;if a Less than A, let a = a +1, so , return to step 20122.

[0022] Optionally, the specific steps of step 20124 are as follows:

[0023] Get the a Events at coordinates The coordinate set of the neighboring points at ;

[0024] Search a The neighboring points in the coordinate set of the event's neighboring points that meet the difference condition are used to project the event image. Perform motion compensation to obtain corrected coordinate values.

[0025] Optionally, the specific steps of step 301 are as follows:

[0026] Step 3011. Original frame image FI Use bilinear interpolation to downsample the resolution and generate downsampled frames with the same resolution as the event camera. ;

[0027] Step 3012. Obtain downsampled frame image Edge gradient image F .

[0028] Optionally, the specific steps of motion compensation in step 20124 are as follows:

[0029] Traverse a All neighboring points of the event, for the k Neighborhood points , the current timestamp mark table coordinate value of the neighborhood point With the a Events at coordinates The coordinate value of The difference in the specified time window When, among them, Indicates the minimum time difference for the same feature to be triggered. Indicates the maximum time difference of the same feature trigger. k Neighborhood points The current coordinate value at Compensation to a The coordinates of the event At , obtain the corrected coordinate value; For the a The event k Neighborhood points x Axis coordinates, For the a The event k Neighborhood points y Axis coordinates.

[0030] Optionally, the expression for the corrected coordinate value is:

[0031]

[0032] in, Indicates the current compensation event image In the a The coordinates of the event Corrected coordinate values.

[0033] Optionally, spatial alignment of a conventional camera and an event camera of an imaging device comprising adjustable optics, a computer, and a signal generator.

[0034] Optionally, the adjustable optical device includes a traditional camera module, an event camera module and a spectroscopic module; the spectroscopic module is arranged between the traditional camera module and the event camera module.

[0035] Optionally, a conventional camera is arranged in the conventional camera module; the conventional camera is arranged to be movable and adjustable in height relative to the spectroscopic module; an event camera is arranged in the event camera module; the event camera is arranged to be movable and adjustable in height relative to the spectroscopic module.

[0036] Compared with the prior art, the present invention has at least the following beneficial effects:

[0037] The advantages of the present application are: one is to provide a motion deblurring event data processing method. By projecting and compensating the event data, the motion deblurring processing of the event frame is carried out, which significantly improves the ability of the event frame to represent the scene details compared with the existing event frame generation method, and provides data basis for improving the sub-pixel level alignment accuracy; the other is to provide a local alignment strategy using pixel-by-pixel and small-range matching, and a semi-dense disparity map is generated by calculating the area correlation in the sliding window as a high-precision alignment result. Compared with the traditional global alignment method of cutting the image to specify the size of the view frame resolution and calculating the similarity of the whole area of the view frame, the local disparity distortion is effectively eliminated, the resolution loss is reduced, and the spatial alignment accuracy is significantly improved. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification, illustrate embodiments of the application and together with the description serve to explain the principles of the application.

[0039] Figure 1 FIG. 1 is a schematic diagram of a camera device of the present application;

[0040] Figure 2 FIG. 2 is a schematic diagram of an adjustable optical device of the present application;

[0041] Figure 3 FIG. 3 is a schematic diagram of a device base platform of the present application;

[0042] Figure 4 FIG. 4 is a schematic diagram of a camera fixing and optical axis alignment method of the present application;

[0043] Figure 5 FIG. 5 is a schematic diagram of the mounting and fixing method of a beam splitter of the present application;

[0044] Figure 6 FIG. 6 is a schematic diagram of a light shielding structure of the present application;

[0045] Figure 7 FIG. 7 is a schematic diagram of a lens mask of different sizes of the present application;

[0046] Figure 8 FIG. 8 is a flowchart of a space-time alignment method of the present application;

[0047] Figure 9 FIG. 9 is a flowchart of a spatial alignment method of the present application.

[0048] REFERENCE NUMERALS:

[0049] 1, traditional camera module; 101, traditional camera; 102, traditional camera adapter plate; 103, first quick-mount module; 104, first sliding block; 2, event camera module; 201, event camera; 202, event camera adapter plate; 203, second quick-mount module; 204, second sliding block; 3, light splitting module; 301, light blocking main body; 302, light splitting mirror base; 303, light splitting mirror top; 304, light splitting mirror; 305, first lens cover; 306, second lens cover; 307, third lens cover; 308, fourth lens cover; 309, lens cover 5; 4, device base platform; 5, computer; 6, signal generator; A, traditional camera sliding groove; B, traditional camera sliding groove; C, event camera sliding groove; D, event camera sliding groove; G, event camera optical axis; E, incident light; F, traditional camera optical axis. DETAILED DESCRIPTION

[0050] In order to enable more clear understanding of the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0051] One embodiment of the present application, see Figure 9 , discloses a space alignment method based on a combination of a traditional camera and an event camera imaging device, which uses the imaging device for synchronous shooting, obtains time-aligned event streams and original frame images according to the space-time alignment method of the traditional camera and the event camera, and performs space alignment of the traditional camera and the event camera based on the original frame images and the event streams, including the following steps:

[0052] Step 101, aligning the traditional camera and the event camera in space to obtain time-synchronized event streams, original frame images and trigger time stamps;

[0053] Step 201, processing the event stream captured by the event camera module 2 and corrected by the trigger time stamp (i.e. after time alignment) to generate a motion deblurred event frame G .

[0054] It can be understood that the event stream captured by the event camera module 2 and corrected by the trigger time stamp includes the event stream of the event camera obtained by the aforementioned space-time alignment method of the traditional camera and the event camera and the event camera correction trigger time stamp.

[0055] Step 2011. Determine the initial time stamp marking table and the initial event image which are the same as the resolution of the event camera.

[0056] The timestamp table is used to store timestamps at different locations, and the event image is used to temporarily store processed data and ultimately generate event frames. The initial timestamp table has all coordinates initialized to 0, and the initial event image has all pixel grayscales initialized to 0.

[0057] Step 2012. Each event in the event stream is sequentially Projecting onto the event image and performing motion compensation on the event image to obtain a compensated event image , the specific steps are:

[0058] Step 20121. Order a =0, when a =0, it is the initial event, the corresponding current timestamp mark table is the initial timestamp mark table, and the current event image is the initial event image;

[0059] Step 20122. Get the first a events The current timestamp marks the coordinates of the table The coordinate value of , the expression is:

[0060]

[0061] in, Respectively represent a Event x Axis coordinates, y Axis coordinates, timestamps, and polarity.

[0062] It is understandable that Mark the table with the current timestamp The timestamp value at the coordinate represents the event camera coordinate system. The timestamp value of the most recent event that occurred at the coordinates.

[0063] Step 20123. a Project the event to the current event image and obtain the projected event image , the expression is:

[0064]

[0065] in, Indicates the a After the event is projected, the event image after projection exist The coordinate value of ; Indicates the current event image exist The coordinate value of .

[0066] Step 20124. a Perform motion compensation on each event to obtain a compensated event image ;

[0067] First, get the a Events at coordinates The coordinate set of the 3×3 neighborhood points at is expressed as:

[0068]

[0069] in, Representing coordinates The coordinate set of the neighboring points at ; Indicates the a The event k The coordinate positions of the neighboring points, k =1,2,…,8.

[0070] Search a The neighboring points in the coordinate set of the event's neighboring points that meet the difference condition are used to project the event image. Perform motion compensation; the specific steps are:

[0071] Traverse a All 8 neighboring points of the event, for the k Neighborhood points , the current timestamp mark table coordinate value of the neighborhood point With the a Events at coordinates The coordinate value of The difference in the specified time window When, among them, Indicates the minimum time difference for the same feature to be triggered. Indicates the maximum time difference of the same feature trigger. k Neighborhood points The current coordinate value at Compensation to a The coordinates of the event At , we get the corrected coordinate value, the expression is:

[0072]

[0073]

[0074] in, Indicates the a After the event is projected, the event image after projection GI' At the neighborhood point The coordinate value of ; Indicates the current compensation event image the coordinate of the first event a the coordinate of the first event the coordinate of the first event the coordinate of the first event the coordinate of the first event k the coordinate of the first event the coordinate of the first event

[0075] Step 20125. judge a the size of the total number of events A, if a greater than or equal to A, the current compensation event image if a less than A, let a = a +1, let , return to step 20122.

[0076] Step 2013. normalize the current compensation event image , generate a motion deblurred event frame G , the expression is:

[0077]

[0078] wherein, the coordinate value of the motion deblurred event frame on the coordinate; the minimum coordinate value in the compensation event image ; the maximum coordinate value in the current compensation event image ; the coordinate value of the current compensation event image on coordinate.

[0079] Step 301, processing the time-aligned original frame image captured by the traditional camera module 1, generating a down-sampling edge gradient image F .

[0080] Step 3011. using bilinear interpolation method to perform resolution down-sampling on the original frame image FI , generating a down-sampling frame picture with the same resolution as the event camera .

[0081] Specifically, determine a frame image with the same resolution as the event camera; for each pixel position of the down-sampling frame picture , calculate its corresponding pixel position FI ( , ), wherein is the frame image is the frame image x is the pixel coordinate on the y axis and is the pixel coordinate on the axis, and the expression is: is the frame image x is the frame image y is the pixel coordinate on the is the pixel coordinate on the

[0082] is the pixel coordinate on the is the pixel coordinate on the is the pixel coordinate on the

[0083] is the pixel coordinate on the is the pixel coordinate on the is the pixel coordinate on the

[0084] is the pixel coordinate on the W f and H f respectively represent the pixel width and the pixel height of the traditional camera, W e and H e respectively represent the pixel width and the pixel height of the event camera.

[0085] For the coordinate point FI in the original frame image , , the first pixel P 1 x 1, y 1), the second pixel P 2 x 2, y 1), the third pixel P 3 x 1, y 2) and the fourth pixel P 4 x 2, y 2) adjacent to it are obtained, and bilinear interpolation is performed, wherein x 1is the pixel coordinate leftmost nearest integer x axis coordinate, x 2is the pixel coordinate rightmost nearest integer x axis coordinate, y 1is the pixel coordinate uppermost nearest integer y axis coordinate, y 2is the pixel coordinate lowermost nearest integer y axis coordinate, to obtain the coordinate value of the frame image after down-sampling at , and the expression is:

[0086]

[0087]

[0088]

[0089] in, I 1 means that in the horizontal direction, the first pixel P 1 and 2 pixels P Interpolation result of 2; I 2 means that in the horizontal direction, for the third pixel P 3 and 4 pixels P Interpolation result of 4; Express I 1 and I 2 The pixel value after vertical interpolation.

[0090] After completing the above steps, a downsampled frame image with the same resolution as the event camera is obtained. .

[0091] Step 3012. Obtain downsampled frame image Edge gradient image F , the expression is:

[0092]

[0093]

[0094]

[0095] in, Represents the frame image after downsampling At pixel position of x Axis direction gradient value; Representing an image At pixel position of y Axis gradient value; Represents the frame image after downsampling In position The pixel value of m and n is the offset, representing the 3×3 neighborhood pixels around the current pixel; G x ( m+ 1, n+ 1) represents the weight value of the horizontal Sobel operator; G y ( m+ 1,n+ 1) represents the weight value of the vertical Sobel operator; Represents the frame image after downsampling In position The edge strength at .

[0096] Step 401: De-motion blurring of event frames based on event cameras G and edge gradient images from traditional cameras F The similarity matching calculation obtains sub-pixel disparity d *.

[0097] Specifically, the edge gradient image of the traditional camera obtained in step 301 F Each pixel coordinate position in , the event frame obtained in step 201 from the event camera with motion blur removed G Center edge parallax search range Sliding Window W , calculate the normalized cross-correlation value, the expression is:

[0098]

[0099] Among them, NCC( d ) indicates the parallax d The normalized cross-correlation value under , indicating the event frame G and edge gradient image F The degree of similarity of local areas; Represents edge gradient image F In coordinates The gray value at Represents edge gradient image F Upper sliding window W The grayscale mean of all pixels in ; Represents an event frame G In coordinates Gray value at ; Represents an event frame G Upper sliding window W The grayscale mean of all pixels in ; d min represents the minimum candidate disparity; d max represents the maximum candidate disparity; Indicates x Pixel offset in the axis direction; Indicates y The pixel offset in the axis direction.

[0100] Furthermore, the value that maximizes NCC is selected as the sub-pixel disparity , the expression is:

[0101] .

[0102] Step 501, calculate sub-pixel level disparity pixel by pixel, and generate an optimized sub-pixel level disparity map, the expression is:

[0103]

[0104] wherein, represents the optimized sub-pixel level disparity value; represents the sub-pixel level disparity; , and respectively represent the NCC value at the sub-pixel level disparity center and its adjacent disparity and .

[0105] Further, when , .

[0106] Further, the spatial alignment of the traditional camera and the event camera of the camera device; synchronously shooting obtains time-synchronized event stream, raw frame image and trigger timestamp.

[0107] Further, referring to Figure 1-Figure 7 , the camera device includes an adjustable optical device, a computer 5 and a signal generator 6; the adjustable optical device includes a traditional camera module 1, an event camera module 2, a light splitting module 3 and a device base platform 4;

[0108] The computer 5 and the signal generator 6 are connected with the traditional camera module 1 and the event camera module 2;

[0109] The light splitting module 3 is arranged between the traditional camera module 1 and the event camera module 2; the traditional camera module 1 and the event camera are arranged at an angle of 90 degrees;

[0110] The traditional camera module 1 includes a first sliding block 104, a first quick-mounting module 103, a traditional camera adapter plate 102 and a traditional camera 101; the first sliding block 104 is slidingly arranged on the device base platform 4, the first quick-mounting module 103 is arranged on the first sliding block 104, the traditional camera adapter plate 102 is arranged on the first quick-mounting module 103, and the traditional camera 101 is arranged on the traditional camera adapter plate 102.

[0111] The event camera module 2 includes a second slider 204, a second quick-release module 203, an event camera adapter plate 202 and an event camera 201; the second slider 204 is slidably set on the device base platform 4, the second quick-release module 203 is set on the second slider 204, the event camera adapter plate 202 is set on the second quick-release module 203, and the event camera 201 is set on the event camera adapter plate 202.

[0112] The spectrometer module 3 includes a light-shielding body 301, a spectrometer base 302, a spectrometer top seat 303, a spectrometer 304, a first lens hood 305 and a second lens hood 306; the spectrometer base 302 is arranged on the device base platform 4, the light-shielding body 301 is arranged on the spectrometer base 302, and the spectrometer top seat 303 is arranged in the through hole at the top of the light-shielding body 301; the spectrometer 304 is fixed by the spectrometer base 302 and the spectrometer top seat 303, and is arranged inside the light-shielding body 301; the first lens hood 305 and the second lens hood 306 are respectively arranged on two adjacent external sides of the light-shielding body 301; the first lens hood 305 is arranged opposite to the traditional camera module 1, and the second lens hood 306 is arranged opposite to the event camera module 2.

[0113] Furthermore, the first slider 104, second slider 204, traditional camera adapter plate 102, event camera adapter plate 202, first quick-release module 103, second quick-release module 203, and base platform 4 of the conventional camera module 1 and event camera module 2 are all made of 6061 aluminum and processed using a CNC machine tool, followed by deburring, rough grinding, and anodizing. These processes are designed to enhance resistance to bending under gravity, improve device stability, reduce surface light reflectivity, and ensure machining accuracy, thereby ensuring accurate optical axis alignment and excellent imaging quality.

[0114] Furthermore, the light shielding body 301, the beam splitter base 302, the beam splitter top 303, the first lens cover 305 and the second lens cover 306 are made of black resin material, processed using stereolithography (SLA) 3D printing technology, and subjected to surface treatment such as deburring and rough grinding.

[0115] Furthermore, the beam splitter 304 uses a K9 glass substrate lens, which has excellent optical transmittance and mechanical stability, and is designed as a 50% transmission and 50% reflection beam splitter to achieve effective and balanced separation of the light path.

[0116] Furthermore, in order to adapt to traditional cameras and event cameras with different optical axis heights, the traditional camera adapter plate 102 and the event camera adapter plate 202 can be replaced with camera adapter plates with different thicknesses and different fixing hole positions.

[0117] Further, see Figure 7To adapt to different lens diameters and avoid the interference of extra light on imaging as much as possible, the first lens cover 305 and the second lens cover 306 can be replaced by the third lens cover 307, the fourth lens cover 308, or the fifth lens cover 309 with different opening diameters; the beam splitter 304 is a 50 / 50 beam splitter.

[0118] Further, the optical axes of the traditional camera 101 and the event camera 201 are in the same plane and perpendicular to each other; the beam splitter 304 is arranged at the intersection of the optical axes of the traditional camera 101 and the event camera 201, and the two mirror surfaces of the beam splitter 304 are arranged at an angle of 45 degrees with the moving direction of the traditional camera 101 and the event camera 201 respectively.

[0119] Further, the traditional camera 101 and the event camera 201 are respectively connected to the computer 5 and the signal generator 6 through data cables.

[0120] Referring to Figure 2-Figure 3 , the traditional camera module 1 and the event camera module 2 use the through holes on the first slider 104 and the second slider 204 respectively to cooperate with the sliding grooves on the device platform 4, and are fixed to the device platform by screws and nuts.

[0121] Further, as Figure 3 indicated, the traditional camera sliding groove combination (including traditional camera sliding grooves A and B) and the event camera sliding groove combination (including traditional camera sliding grooves C and D) are perpendicular to each other, keeping the optical axes of the traditional camera and the event camera always perpendicular to each other.

[0122] Further, when replacing lenses of different lengths and aligning the field of view size, the motion of the traditional camera module 1 and the event camera module 2 is respectively limited to a straight line, ensuring that the incident light E passes through the transmission / reflection of the beam splitter 304 and can be accurately aligned with the event camera optical axis G and the traditional camera optical axis F.

[0123] Specifically, the first slider 104 and the second slider 204 are provided with six through holes for fixing, which are fixed to the device base platform 4 by screws and nuts. The screws can also function as positioning pins to limit the positions of the two sliders.

[0124] Referring to Figure 4 , by replacing the traditional camera adapter plate 102 and the event camera adapter plate 202 with different thicknesses, the optical axes of the traditional camera and the event camera can be adjusted to the same specified height plane, achieving horizontal alignment of the optical axes.

[0125] Further, the thickness of the traditional camera adapter plate 102 is:

[0126]

[0127] Thickness of the event camera adapter plate 202 t 2 is:

[0128]

[0129] in, h is the optical axis height of the camera device; is the sum of the thickness of the slider and its corresponding quick-release module (i.e., the sum of the thickness of the first slider 104 and the first quick-release module 103 is the same as the sum of the thickness of the second slider 204 and the second quick-release module 203); 、 are the optical axis heights of the traditional camera and event camera, respectively.

[0130] See also Figure 3 and Figure 5 The device's base platform 4 features a 45-degree mounting slot at the intersection of the perpendicular symmetric centers of the conventional camera slot assembly (including conventional camera slots A and B) and the event camera slot assembly (including conventional camera slots C and D). This slot is used to secure the beam splitter base 302. The beam splitter top mount 303 is secured to the square through-hole of the light shielding body 301. Both the beam splitter top mount 303 and the base 302 have square slots for securing the beam splitter 304. Once secured, the beam splitter 304 forms a 45-degree angle with the optical axes of the conventional camera and the event camera, equally dividing the incident light. This securement utilizes an interference fit.

[0131] See also Figure 6 Furthermore, a light shielding body 301, a first lens hood 305, and a second lens hood 306 are used to surround the beam splitter 304 to prevent interference such as ghosting and reflections. The light shielding body 301, the first lens hood 305, and the second lens hood 306 are made of low-reflective material and are secured together by an interference fit.

[0132] Further, see Figure 8 In step 201, the event stream captured by the event camera and corrected for the trigger timestamp (i.e., after time alignment) is processed using a spatiotemporal alignment method for a traditional camera and an event camera, including the following steps:

[0133] Step 1. Use the aforementioned camera device to perform spatiotemporal alignment to obtain the event stream of the event camera and the original frame image of the traditional camera.

[0134] Specifically, data acquisition is performed using a conventional camera module 1 with adjustable optical devices, an event camera module 2, and a spectrometer module 3. Conventional camera module 1 captures images and sends the captured raw frame images to computer 5; while event camera module 2 captures images and sends the captured event streams to computer 5.

[0135] As can be understood, the adjustable optical setup is used for spatial alignment. When acquiring data using the adjustable optical setup, the spatial alignment of the conventional camera and the event camera minimizes differences between the two cameras in terms of their optical axis direction, imaging plane, and overlapping viewing angle.

[0136] Step 2. Obtain trigger timestamp data from traditional cameras and event cameras.

[0137] Specifically, when the adjustable optical device is used for shooting, the signal generator 6 is used to output a periodic square wave signal, and the rising edge signal of the square wave signal is used as a trigger signal; the trigger signal is distributed to the traditional camera and the event camera at the same time through a low-latency distribution cable, and the time interval between the two trigger signals is .

[0138] Preferably, the periodic square wave signal output by the signal generator 6 has a low level of 0V and a high level of 5V, conforming to the TTL (Transistor-Transistor Logic) digital circuit standard, and a duty cycle of 50%. The transition of the square wave signal from a low level to a high level (i.e., a rising edge) in each cycle is used as a trigger signal to trigger traditional cameras and event cameras.

[0139] It can be understood that the time interval between two trigger signals (i.e. the period of the square wave signal) is recorded as , which is determined by the trigger frequency. Specifically, the event camera detects the trigger signal and obtains the first i Trigger timestamp When the traditional camera detects the rising edge signal, it starts to capture the image and obtains the first j Trigger timestamp ,in, n The total number of timestamps for event camera triggers, m is the total number of trigger timestamps of the traditional camera. Both the traditional camera and the event camera respond to the trigger signal through the BNC interface, obtain the corresponding trigger timestamp, and send the trigger timestamp to the computer 5 through the data line.

[0140] Step 3. Computer 5 determines the modified threshold value based on the trigger frequency and the trigger timestamp interval of the event camera Trigger timestamp for event camera Perform smoothing correction and obtain the event camera correction trigger timestamp.

[0141] Furthermore, .

[0142] Furthermore, the trigger timestamp interval of the event camera The expression is:

[0143]

[0144] in, For event cameras i A trigger timestamp; For event cameras A trigger timestamp; , n The total number of timestamps for event camera triggers.

[0145] Furthermore, if , indicating that the event camera may respond to the same trigger signal multiple times. The trigger timestamp of the event camera is To correct the trigger timestamp for false triggering data, the expression for correcting the timestamp is:

[0146]

[0147] in, Indicates the event camera i Trigger timestamp Correction value of represents the arithmetic mean function; Indicates the event camera A trigger timestamp to be corrected, , Indicates the event camera k A trigger timestamp to be corrected, , No. k Trigger timestamps to be corrected The timestamp of the last pending revision trigger.

[0148] Furthermore, when and When the camera triggers the event, it indicates the timestamp To effectively trigger the data, the event camera trigger timestamp is used as the event camera corrected trigger timestamp.

[0149] Furthermore, when and When the event camera responds to the trigger signal incorrectly, such as incorrectly identifying the rising edge and falling edge, the event camera trigger timestamp To correct the false trigger data, the timestamp is corrected and the timestamp interval is corrected using the cubic spline interpolation algorithm. The error is smoothed and corrected to obtain the corrected timestamp. The expression is:

[0150]

[0151] in, Indicates the event camerai Trigger timestamp Correction value of Indicates the first trigger timestamp of the event camera (i.e., the start time of the trigger time series); Indicates the time interval between two trigger signals; For event cameras q The error correction term is obtained by spline interpolation of the trigger timestamp. The interpolation is based on the interval of.

[0152] After completing the above steps, the number of valid trigger event stamps of the event camera is corrected to n ', and the number of trigger timestamps of traditional cameras m And the actual number of trigger signals N Stay consistent and satisfy .

[0153] Step 4. Establish the trigger timestamp mapping relationship between the traditional camera and the event camera, map the corrected trigger timestamp of the event camera to the trigger timestamp reference system of the traditional camera, and obtain the trigger timestamp of the event camera after mapping. The expression is:

[0154]

[0155] in, Represents the first event camera after mapping i A trigger timestamp; represents the clock rate scaling factor; Indicates the clock offset; N Indicates the actual number of trigger signals.

[0156] Furthermore, the clock rate scaling factor and clock offset The expression is:

[0157]

[0158]

[0159] in, Indicates the first j A trigger timestamp, , m Total number of trigger timestamps for legacy cameras.

[0160] Step 5. Based on the trigger timestamp of the mapped event camera, obtain the time synchronization error between the traditional camera and the event camera , the expression is:

[0161] .

[0162] Step 6. Based on the time synchronization error between the traditional camera and the event camera, obtain the mean deviation of the synchronization error, which is expressed as:

[0163]

[0164] in, represents the mean deviation of the time synchronization error between the traditional camera and the event camera; Represents the time synchronization error between the traditional camera and the event camera.

[0165] Step 7. Use the mean deviation of the synchronization error to align the timestamps of the traditional camera to obtain the aligned timestamps of the traditional camera , the expression is:

[0166] .

[0167] After the above steps, the traditional camera and the event camera can be aligned in time, that is, the timestamps of the original frame image of the traditional camera and the event stream data of the event camera are kept correctly corresponding.

[0168] The spatiotemporal alignment camera device and method of the present invention can simultaneously capture spatiotemporal alignment of original frame images and event stream data, and correct the corresponding trigger timestamp data. During the external trigger process, a single trigger signal may be responded to multiple times in a short period of time, and the intervals between these timestamps are often too small. In addition, the event camera may incorrectly identify the rising edge or falling edge, resulting in the trigger timestamp interval being too small and not in line with the theoretical value. The difference is large, that is, the timestamp interval distribution is uneven. The method of the present invention can avoid the influence of the above-mentioned signal jitter or noise on the camera data acquisition.

[0169] The present invention's imaging device and method for spatiotemporal alignment of a conventional camera and an event camera utilizes an adjustable optical device for coarse spatial alignment, ensuring the two cameras' optical axis directions, imaging planes, and overlapping viewing angles are as consistent as possible. A signal generator triggers temporal alignment, correcting for temporal synchronization errors to ensure the two cameras capture images or events at the same time. By generating motion-blur-eliminating event frames and optimizing sub-pixel parallax, high-precision pixel-level spatial alignment of the event camera and conventional camera is achieved. The spatiotemporal alignment of the conventional and event camera data can be used for modal fusion, such as robotic navigation, object detection, 3D reconstruction, environmental perception, and other subsequent tasks.

[0170] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. A spatial alignment method based on a combined imaging device of a traditional camera and an event camera, characterized in that: The steps include: Step 101: spatially aligning a conventional camera and an event camera to obtain an event stream captured by the time-synchronized event camera and corrected by trigger timestamps, and a raw frame image captured by the conventional camera and corrected by time alignment and trigger timestamps. Step 201: Process the event stream to generate a motion-blurred event frame. The specific steps are as follows: Step 2011. Determine an initial timestamp tag table and an initial event image with the same resolution as the event camera; Step 2012. Sequentially a events Projecting onto the event image and performing motion compensation on the event image to obtain a compensated event image ,in, Respectively represent a Event x Axis coordinates, y axis coordinates and timestamps; Step 2013. Compensate event image Perform normalization to generate motion-blurred event frames G ; Among them, the specific steps of step 2012 are as follows: Step 20121. Order a =0, when a =0, it is the initial event, the corresponding current timestamp mark table is the initial timestamp mark table, and the current event image is the initial event image; Step 20122. Get the first a events The current timestamp marks the coordinates of the table The coordinate value of ; Step 20123. a Project the event to the current event image and obtain the projected event image ; Step 20124. a Perform motion compensation on each event to obtain the current compensated event image ; Step 20125. Judgment a With the size of the total number of events A, if a Greater than or equal to A, then use the current compensation event image ;if a Less than A, let a = a +1, so , return to step 20122; Step 301, processing the original frame image to generate a downsampled edge gradient image; Step 401: De-motion blurring of event frames based on event cameras G and edge gradient images from traditional cameras F The similarity matching calculation obtains sub-pixel disparity; Step 501 : Calculate sub-pixel disparity pixel by pixel and generate an optimized sub-pixel disparity map.

2. The spatial alignment method according to claim 1, characterized in that: The specific steps of step 20124 are as follows: Get the a Events at coordinates The coordinate set of the neighboring points at ; Search a The neighboring points in the coordinate set of the event's neighboring points that meet the difference condition are used to project the event image. Perform motion compensation to obtain corrected coordinate values.

3. The spatial alignment method according to claim 1, characterized in that: The specific steps of step 301 are as follows: Step 3011. Original frame image FI Use bilinear interpolation to downsample the resolution and generate downsampled frames with the same resolution as the event camera. ; Step 3012. Obtain downsampled frame image Edge gradient image F .

4. The spatial alignment method according to claim 2, characterized in that: The specific steps of motion compensation in step 20124 are as follows: Traverse a All neighboring points of the event, for the k Neighborhood points , the current timestamp mark table coordinate value of the neighborhood point With the a Events at coordinates The coordinate value of The difference in the specified time window When, among them, Indicates the minimum time difference for the same feature to be triggered. Indicates the maximum time difference of the same feature trigger. k Neighborhood points The current coordinate value at Compensation to a The coordinates of the event At , obtain the corrected coordinate value; For the a The event k Neighborhood points x Axis coordinates, For the a The event k Neighborhood points y Axis coordinates.

5. The spatial alignment method according to claim 4, characterized in that: The expression of the corrected coordinate value is: in, Indicates the current compensation event image In the a The coordinates of the event Corrected coordinate values.

6. The spatial alignment method according to any one of claims 1 to 5, characterized in that: Spatial alignment of a conventional camera and an event camera of an imaging setup comprising adjustable optics, a computer, and a signal generator.

7. The spatial alignment method according to claim 6, characterized in that: The adjustable optical device includes a traditional camera module, an event camera module and a spectroscopic module; the spectroscopic module is arranged between the traditional camera module and the event camera module.

8. The spatial alignment method according to claim 7, characterized in that: The traditional camera is arranged in the traditional camera module; the traditional camera is arranged to be movable and adjustable relative to the spectroscopic module; the event camera is arranged in the event camera module; the event camera is arranged to be movable and adjustable relative to the spectroscopic module.

Images