A photographing apparatus and method for spatio-temporal alignment of a traditional camera and an event camera

Abstract

The present application relates to a kind of traditional camera and event camera space-time alignment camera and method, belong to camera technology field, solve the technical problem of low time synchronization accuracy in prior art;Space alignment mechanical structure solidification;With the technical problem of missing space-time alignment method of collaborative spectroscopy system.The traditional camera and event camera space-time alignment camera of the present application includes adjustable optical device, computer and signal generator;Adjustable optical device includes traditional camera module, event camera module, spectroscopy module and device base platform;Traditional camera module and event camera are arranged at 90 degrees;The spectroscopy module includes beam splitter, and the beam splitter is arranged between traditional camera module and event camera module, and the sensor, lens configuration and its installation position of portable adjustable use can be achieved accurate space-time alignment of two cameras by space-time alignment method, with high flexibility, high precision and other characteristics.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of camera devices, in particular to a camera device and method for spatio-temporal alignment of a traditional camera and an event camera. BACKGROUND

[0002] Traditional cameras are based on the principle of frame-by-frame acquisition, which accumulates light signals within the exposure time of each frame and converts them into digital images to record the visual information of the scene in image frames, which is also referred to as an ordinary industrial camera in the present application. However, traditional cameras have many serious challenges in terms of output frequency, dynamic range, power consumption, and delay. To solve these limitations, event cameras (dynamic vision sensors) have been successfully developed, which asynchronously sample the intensity changes of each pixel with fine temporal resolution, generating an asynchronous event stream, which in turn provides the advantages of high temporal resolution, high dynamic range, low delay, and low power consumption. Therefore, the limited applicability of each individual sensor has led to the need for sensor fusion technology to enable these sensors to complement each other's limitations. Existing research shows that the fusion of traditional cameras and event cameras provides practical solutions for high-quality image reconstruction, accurate estimation of pixel-level motion in challenging scenes, and efficient detection and tracking of targets for downstream tasks. However, the prerequisite for the fusion of data from traditional cameras and event cameras is to obtain accurate spatio-temporal correspondence.

[0003] Currently, for the temporal alignment of the heterogeneous traditional camera and event camera in a camera device equipped with both, software algorithms are commonly used for synchronization, resulting in low temporal alignment accuracy. For the spatial alignment of the two, common methods include using a beam splitter for alignment or binocular camera disparity matching. Binocular heterogeneous data matching relies on complex computational processing, which has high computational cost, and is sensitive to environmental conditions, making it prone to matching errors or failure in areas with insufficient texture or occlusions. The beam splitter allows the traditional camera and event camera to share the same optical path, simplifying the alignment process and making the data from the traditional camera and event camera more easily fused, which is particularly suitable for applications that require a combination of high temporal resolution and high spatial resolution, such as efficient frame interpolation, real-time visual enhancement, and accurate motion detection. However, existing beam splitting devices are typically designed as fixed structures, do not support flexible combination and replacement of the lenses of the traditional camera and event camera, limiting the flexibility and scalability of the system, and lack a corresponding spatio-temporal alignment method. SUMMARY

[0004] The present application aims to address the deficiencies in the spatio-temporal alignment of two cameras in current camera systems, and proposes a camera device and method for spatio-temporal alignment of a traditional camera and an event camera, which can conveniently adjust the used sensors, lens configurations, and their installation positions, and achieve accurate spatio-temporal alignment of the two cameras through a spatio-temporal alignment method, with high flexibility, high precision, and other characteristics.

[0005] The application provides a camera device for spatio-temporal alignment of traditional camera and event camera, comprising adjustable optical device, device base platform, computer and signal generator.

[0006] The adjustable optical device comprises traditional camera module, event camera module, light splitting module and device base platform.

[0007] The computer and the signal generator are connected with the traditional camera module and the event camera module.

[0008] The traditional camera module comprises traditional camera; the traditional camera is movably arranged on the device base platform.

[0009] The event camera module comprises event camera; the event camera is movably arranged on the device base platform.

[0010] The traditional camera and the event camera are arranged at an angle of 90 degrees.

[0011] The light splitting module comprises light splitting mirror; the light splitting mirror is arranged between the traditional camera module and the event camera module.

[0012] Optionally, the light splitting module further comprises light shielding body, first lens cover and second lens cover; the light splitting mirror is arranged in the light shielding body; the first lens cover and the second lens cover are arranged on two adjacent outer sides of the light shielding body respectively; the first lens cover is arranged opposite to the traditional camera module, and the second lens cover is arranged opposite to the event camera module.

[0013] Optionally, the optical axes of the traditional camera and the event camera are in the same plane and perpendicular to each other.

[0014] Optionally, the light splitting mirror is arranged at the intersection of the optical axes of the traditional camera and the event camera.

[0015] In another aspect of the application, a method for spatio-temporal alignment of traditional camera and event camera is disclosed, and the specific steps are as follows:

[0016] Step 1. Obtain event stream of event camera and original frame image of traditional camera.

[0017] Step 2. Obtain trigger timestamp data of traditional camera and event camera.

[0018] Step 3. According to the correction threshold determined by trigger frequency and the trigger timestamp interval of event camera Smoothly correct the trigger timestamp of event camera to obtain the corrected trigger timestamp of event camera.

[0019] Step 4. Establish the trigger timestamp mapping relationship between the traditional camera and the event camera, map the modified trigger timestamp of the event camera to the trigger timestamp reference system of the traditional camera, and obtain the trigger timestamp of the event camera after mapping;

[0020] Step 5. Based on the trigger timestamp of the mapped event camera, obtain the time synchronization error between the traditional camera and the event camera;

[0021] Step 6. Based on the time synchronization error of the traditional camera and the event camera, obtain the mean deviation of the synchronization error;

[0022] Step 7. Align the traditional camera timestamps using the mean deviation of the synchronization error to obtain the aligned traditional camera timestamps.

[0023] Optionally, the criterion for smoothly correcting the trigger timestamp of the event camera based on the correction threshold determined by the trigger frequency and the trigger timestamp interval of the event camera is as follows:

[0024] if The first event camera i Trigger timestamp The trigger timestamp has been corrected as the data was triggered erroneously.

[0025] if and The first event camera i Trigger timestamp To ensure effective data triggering, the event camera trigger timestamp is used as the event camera correction trigger timestamp; Indicates the time interval between two trigger signals;

[0026] if and The first event camera i Trigger timestamp This timestamp has been corrected to correct data that was mistakenly triggered.

[0027] Optionally, the expression for the trigger timestamp of the mapped event camera is:

[0028]

[0029] in, Represents the first event camera after mapping i One trigger timestamp; Indicates the clock rate scaling factor; Indicates the clock offset; Indicates the event camera number i Trigger timestamp Correction value; N This indicates the actual number of trigger signals.

[0030] Optionally, the clock rate scale factor is expressed as:

[0031]

[0032] wherein, denotes the i-th trigger timestamp of the traditional camera. j

[0033] Optionally, the clock offset is expressed as:

[0034]

[0035] wherein, denotes the i-th trigger timestamp of the traditional camera. j

[0036] In a third aspect, the application discloses a method for spatial alignment of a traditional camera and an event camera, and the specific steps are as follows: obtaining the time-aligned event stream and the original frame image according to the aforementioned method for spatial-temporal alignment of a traditional camera and an event camera, and performing spatial alignment of the traditional camera and the event camera based on the original frame image and the event stream.

[0037] Compared with the prior art, the application has at least the following beneficial effects: the application has the advantages that the traditional camera and the event camera are designed through an optical system and the like, a space coarse alignment is performed through an adjustable optical device, the consistency of the optical axis direction, the imaging plane, the visual angle overlapping area and the like of the two cameras is ensured, and the flexibility is high. A spatial-temporal alignment method with software and hardware is provided, the time synchronization of the two cameras is ensured, the high-precision spatial-temporal alignment of the traditional camera and the event camera is realized, and a good foundation is provided for data fusion under a high dynamic range scene and high-speed motion capture. BRIEF DESCRIPTION OF DRAWINGS

[0038] The accompanying drawings are included to provide a further understanding of the application and are incorporated in and constitute a part of this specification. The drawings illustrate embodiments of the application and, together with the description, serve to explain the principles of the application.

[0039] Figure 1 FIG. 1 is a schematic diagram of a camera device according to the application;

[0040] Figure 2 FIG. 2 is a schematic diagram of an adjustable optical device according to the application;

[0041] Figure 3 FIG. 3 is a schematic diagram of a device base platform according to the application;

[0042] Figure 4 FIG. 4 is a schematic diagram of a camera fixing and optical axis alignment method according to the application;

[0043] ​​Figure 5 Figure 1 is a schematic diagram of the mounting and fixing method of the spectroscope of the present application;

[0044] Figure 6 Figure 2 is a schematic diagram of the light shielding structure of the present application;

[0045] Figure 7 Figure 3 is a schematic diagram of the lens mask of different sizes of the present application;

[0046] Figure 8 Figure 4 is a flow chart of the space-time alignment method of the present application;

[0047] Figure 9 Figure 5 is a flow chart of the space alignment method of the present application.

[0048] Explanation of reference signs:

[0049] 1, traditional camera module; 101, traditional camera; 102, traditional camera adapter plate; 103, first quick-mount module; 104, first sliding block; 2, event camera module; 201, event camera; 202, event camera adapter plate; 203, second quick-mount module; 204, second sliding block; 3, light splitting module; 301, light shielding main body; 302, spectroscope base; 303, spectroscope top seat; 304, spectroscope; 305, first lens mask; 306, second lens mask; 307, third lens mask; 308, fourth lens mask; 309, fifth lens mask; 4, device base platform; 5, computer; 6, signal generator; A, traditional camera sliding groove; B, traditional camera sliding groove; C, event camera sliding groove; D, event camera sliding groove; G, event camera optical axis; E, incident light; F, traditional camera optical axis. DETAILED DESCRIPTION

[0050] In order to enable a clearer understanding of the above-mentioned purposes, features and advantages of the present application, the present application will be further described in detail below in combination with the drawings and specific embodiments. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict. In addition, the present application can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present application is not limited by the specific embodiments disclosed below.

[0051] One embodiment of the present application, see Figures 1-7 , discloses a kind of traditional camera and event camera space-time alignment camera device, including adjustable optical device, computer 5 and signal generator 6;

[0052] Adjustable optical device includes traditional camera module 1, event camera module 2, light splitting module 3 and device base platform 4;

[0053] The computer 5 and the signal generator 6 are connected with the traditional camera module 1 and the event camera module 2;

[0054] The light splitting module 3 is arranged between the traditional camera module 1 and the event camera module 2; the traditional camera module 1 and the event camera module 2 are arranged at an angle of 90 degrees;

[0055] The traditional camera module 1 comprises a first sliding block 104, a first quick-mount module 103, a traditional camera adapter plate 102 and a traditional camera 101; the first sliding block 104 is slidingly arranged on the device base platform 4, the first quick-mount module 103 is arranged on the first sliding block 104, the traditional camera adapter plate 102 is arranged on the first quick-mount module 103, and the traditional camera 101 is arranged on the traditional camera adapter plate 102.

[0056] The event camera module 2 comprises a second sliding block 204, a second quick-mount module 203, an event camera adapter plate 202 and an event camera 201; the second sliding block 204 is slidingly arranged on the device base platform 4, the second quick-mount module 203 is arranged on the second sliding block 204, the event camera adapter plate 202 is arranged on the second quick-mount module 203, and the event camera 201 is arranged on the event camera adapter plate 202.

[0057] The light splitting module 3 comprises a light shielding main body 301, a light splitting mirror base 302, a light splitting mirror top base 303, a light splitting mirror 304, a first lens cover 305 and a second lens cover 306; the light splitting mirror base 302 is arranged on the device base platform 4, the light shielding main body 301 is arranged on the light splitting mirror base 302, and the light splitting mirror top base 303 is arranged on a through hole at the top of the light shielding main body 301; the light splitting mirror 304 is fixed by the light splitting mirror base 302 and the light splitting mirror top base 303 and arranged inside the light shielding main body 301; the first lens cover 305 and the second lens cover 306 are arranged on two adjacent outer sides of the light shielding main body 301 respectively; the first lens cover 305 is arranged opposite to the traditional camera module 1, and the second lens cover 306 is arranged opposite to the event camera module 2.

[0058] Further, referring to Figure 7 To adapt to different lens diameters and avoid interference of excess light on imaging as much as possible, the first lens cover 305 and the second lens cover 306 can be replaced by a third lens cover 307, a fourth lens cover 308 or a fifth lens cover 309 with different opening diameters; the light splitting mirror 304 is a 50 / 50 light splitting mirror.

[0059] Further, the optical axes of the traditional camera 101 and the event camera 201 are in the same plane and perpendicular to each other; the light splitting mirror 304 is arranged at the intersection of the optical axes of the traditional camera 101 and the event camera 201, and the two mirror surfaces of the light splitting mirror 304 are arranged at an angle of 45 degrees with the moving directions of the traditional camera 101 and the event camera 201 respectively.

[0060] Further, the conventional camera 101 and the event camera 201 are connected to the computer 5 and the signal generator 6 respectively through data cables.

[0061] Referring to Figures 2-3 , the conventional camera module 1 and the event camera module 2 are fixed to the device base platform 4 by using the through holes on the first slider 104 and the second slider 204 to cooperate with the sliding grooves on the device base platform 4, and by using screws and nuts.

[0062] Further, as shown in Figure 3 , the conventional camera sliding groove combination (including the conventional camera sliding grooves A and B) and the event camera sliding groove combination (including the conventional camera sliding grooves C and D) are perpendicular to each other, so as to keep the optical axes of the conventional camera and the event camera always perpendicular to each other.

[0063] Further, when changing the lenses of different lengths and aligning the field of view size, the movements of the conventional camera module 1 and the event camera module 2 are respectively limited to a straight line, so as to ensure that the incident light E passes through the transmission / reflection of the beam splitter 304 and can be accurately aligned with the optical axis G of the event camera and the optical axis F of the conventional camera.

[0064] Specifically, the first slider 104 and the second slider 204 are provided with six through holes for fixing, and are fixed to the device base platform 4 by using screws and nuts. The screws can also function as positioning pins to limit the positions of the two sliders.

[0065] Referring to Figure 4 , by replacing the conventional camera adapter plate 102 and the event camera adapter plate 202 of different thicknesses, the optical axes of the conventional camera and the event camera can be adjusted to the same specified height plane, so as to realize the horizontal alignment of the optical axes.

[0066] Further, the thickness of the conventional camera adapter plate 102 is:

[0067]

[0068] The thickness of the event camera adapter plate 202 is: t 2:

[0069]

[0070] wherein, h is the height of the optical axis of the camera device; is the sum of the thicknesses of the slider and the corresponding quick-mount module (i.e., the sum of the thicknesses of the first slider 104 and the first quick-mount module 103, and the sum of the thicknesses of the second slider 204 and the second quick-mount module 203); , are the heights of the optical axes of the conventional camera and the event camera respectively.

[0071] Referring to Figure 3 and Figure 5 , the device base platform 4 sets a 45-degree mounting groove at the intersection of the symmetry centers of the traditional camera sliding groove combination and the event camera sliding groove combination, for fixing the beam splitter base 302; the beam splitter top base 303 is fixed on the square through hole of the light shielding main body 301. The beam splitter top base 303 and the beam splitter base 302 are each provided with a square groove for fixing the beam splitter 304. After the beam splitter 304 is fixed, it forms a 45-degree angle with the two optical axes of the traditional camera and the event camera, and equally divides the incident light. The above-mentioned fixing uses interference fit.

[0072] Referring to Figure 6 , further, the beam splitter 304 is surrounded by the light shielding main body 301, the first lens cover 305 and the second lens cover 306 to avoid interference such as ghosting and reflection. The light shielding main body 301, the first lens cover 305 and the second lens cover 306 are made of low-reflective material. They are fixed by interference fit.

[0073] On the other hand, referring to Figure 8 , the present application proposes a method for spatio-temporal alignment of a traditional camera and an event camera, which uses the aforementioned camera device for spatio-temporal alignment, including the following steps:

[0074] Step 1. Obtain the event stream of the event camera and the raw frame image of the traditional camera.

[0075] Specifically, the adjustable optical device, the traditional camera module 1, the event camera module 2 and the beam splitter module 3 are used for data acquisition. The traditional camera module 1 is used for shooting, and the captured raw frame image is sent to the computer 5; at the same time, the event camera module 2 is used for shooting, and the captured event stream is sent to the computer 5.

[0076] It can be understood that the adjustable optical device is used for spatial alignment. When the adjustable optical device is used for data acquisition, the spatial alignment of the traditional camera and the event camera minimizes the differences in the optical axis direction, the imaging plane and the overlapping area of the view angle of the two cameras.

[0077] Step 2. Obtain the trigger timestamp data of the traditional camera and the event camera.

[0078] Specifically, when the adjustable optical device is used for shooting, a periodic square wave signal is output by the signal generator 6, and the rising edge signal of the square wave signal is used as the trigger signal; the trigger signal is distributed to the traditional camera and the event camera through a low-delay distribution cable, and the time interval between the two trigger signals is .

[0079] Preferably, the periodic square wave signal outputted by the signal generator 6 has a low level of 0 V and a high level of 5 V, which meets the TTL (Transistor-Transistor Logic) digital circuit standard, and a duty cycle of 50%, and the square wave signal jumps from the low level to the high level (i.e. the rising edge) in each period is taken as the trigger signal for triggering the traditional camera and the event camera.

[0080] It can be understood that the time interval between the two trigger signals (i.e. the period of the square wave signal) is denoted as , which is determined by the trigger frequency. Specifically, the event camera detects the trigger signal and obtains the i th trigger timestamp of the event camera, and the traditional camera starts image acquisition when detecting the rising edge signal and obtains the j th trigger timestamp of the traditional camera, where n is the total number of trigger timestamps of the event camera, m is the total number of trigger timestamps of the traditional camera. Both the traditional camera and the event camera respond to the trigger signal through the BNC interface, obtain the response trigger timestamp, and send the trigger timestamp to the computer 5 through the data line.

[0081] Step 3. The computer 5 performs a smoothing correction on the trigger timestamps of the event camera according to the correction threshold determined according to the trigger frequency and the trigger timestamp interval of the event camera to obtain the corrected trigger timestamps of the event camera.

[0082] Further, .

[0083] Further, the expression of the trigger timestamp interval of the event camera is:

[0084]

[0085] where is the i th trigger timestamp of the event camera; is the th trigger timestamp of the event camera; , n is the total number of trigger timestamps of the event camera.

[0086] Further, if , it indicates that the event camera may respond to the same trigger signal multiple times, and the trigger timestamp of this event camera is false trigger data. The expression for correcting the trigger timestamp is: ​

[0087]

[0088] wherein, denotes the correction value of the event camera's i th trigger timestamp; denotes the arithmetic mean function; denotes the event camera's th trigger timestamp to be corrected, , denotes the event camera's k th trigger timestamp to be corrected, , the k th trigger timestamp to be corrected is the last trigger timestamp to be corrected.

[0089] Further, when and , it indicates that the event camera's trigger timestamp is valid trigger data, and the event camera's trigger timestamp is taken as the event camera's corrected trigger timestamp.

[0090] Further, when and , it indicates that the event camera may have responded to the trigger signal incorrectly, such as incorrectly identifying the rising edge and the falling edge, and the event camera's trigger timestamp is false trigger data. The timestamp is corrected by using a cubic spline interpolation algorithm to smooth the error of the timestamp interval , and a corrected timestamp is obtained, expressed as:

[0091]

[0092] wherein, denotes the correction value of the event camera's i th trigger timestamp; denotes the event camera's th trigger timestamp (i.e., the starting time of the trigger time sequence); denotes the time interval between two trigger signals; is the error correction term of the event camera's q th trigger timestamp obtained by spline interpolation, and the interpolation is based on the interval .

[0093] After the above steps are completed, the number of valid trigger timestamps of the event camera is corrected to , which is consistent with the number of trigger timestamps of the traditional camera m and the number of real trigger signals N , and satisfies​ .

[0094] Step 4. Establish the trigger timestamp mapping relationship between the traditional camera and the event camera, correct the trigger timestamp of the event camera into the trigger timestamp reference system of the traditional camera, and obtain the trigger timestamp of the mapped event camera, expressed as:

[0095]

[0096] wherein, represents the i-th trigger timestamp of the mapped event camera; i represents the clock rate scale factor; represents the clock offset; represents the actual number of trigger signals. N Further, the expression of the clock rate scale factor

[0097] and the clock offset is:

[0098]

[0099]

[0100] wherein, represents the i-th trigger timestamp of the traditional camera, j , is the total number of trigger timestamps of the traditional camera. m

[0101] Step 5. Based on the trigger timestamp of the mapped event camera, obtain the time synchronization error between the traditional camera and the event camera, expressed as:

[0102] .

[0103] Step 6. Based on the time synchronization error between the traditional camera and the event camera, obtain the mean deviation of the synchronization error, expressed as:

[0104]

[0105] wherein, represents the mean deviation of the time synchronization error between the traditional camera and the event camera; represents the time synchronization error between the traditional camera and the event camera.

[0106] Step 7. Align the time stamp of the traditional camera using the mean deviation of the synchronization error, and obtain the aligned time stamp of the traditional camera , expressed as:

[0107] ​​ .

[0108] After the above steps, the traditional camera and the event camera can be time-aligned, that is, the time stamps of the traditional camera original frame image and the event stream data of the event camera are correctly corresponded.

[0109] The space-time aligned camera and method of the present application can simultaneously capture the space-time aligned original frame image and event stream data, and correct the corresponding trigger time stamp data. In the external trigger process, the camera may respond to a single trigger signal multiple times in a short time, and the time stamps are often too small. In addition, the event camera may incorrectly identify the rising edge or falling edge, resulting in a large difference between the trigger time stamp interval and the theoretical difference, that is, the time stamp interval is not uniform. The method of the present application can avoid the influence of the above signal jitter or noise on the camera data acquisition.

[0110] In a third aspect of the present application, referring to Figure 9 , a space alignment method of a traditional camera and an event camera is disclosed, which uses the aforementioned camera to take pictures synchronously, acquires time-aligned event stream and original frame image according to the aforementioned space-time alignment method of the traditional camera and the event camera, and performs space alignment of the traditional camera and the event camera based on the original frame image and the event stream, including the following steps:

[0111] Step 101, using the adjustable optical device of the aforementioned camera, the traditional camera module 1, the event camera module 2 and the light splitting module 3 to perform space alignment; synchronously taking pictures to obtain time-synchronized event stream, original frame image and trigger time stamp;

[0112] Step 201, processing the event stream captured by the event camera module 2 and corrected by the trigger time stamp (i.e. time-aligned) to generate a motion deblurred event frame G .

[0113] It can be understood that the event stream captured by the event camera module 2 and corrected by the trigger time stamp includes the event stream of the event camera obtained by the aforementioned space-time alignment method of the traditional camera and the event camera and the corrected trigger time stamp of the event camera.

[0114] Step 2011. Determine an initial time stamp marker table and an initial event image with the same resolution as the event camera.

[0115] The time stamp marker table is used to save the time stamp markers at different positions, and the event image is used to temporarily store processing data and finally used to generate an event frame. The initial value of all coordinates of the initial time stamp marker table is 0; the initial value of the pixel gray scale of the initial event image is 0.

[0116] Step 2012. Sequentially mark each event in the event stream with the initial time stamp marker table. Projecting the event to the event image and motion compensating the event image to obtain a compensated event image The specific steps are as follows:

[0117] Step 20121. Let a =0, when a =0, it is an initial event, the corresponding current timestamp marker table is an initial timestamp marker table, and the current event image is an initial event image;

[0118] Step 20122. Obtain the coordinate value of the coordinate a of the current timestamp marker table of the th event in the event stream , expressed as:

[0119]

[0120] Wherein, respectively represent the a th event x axis coordinate, y axis coordinate, timestamp and polarity.

[0121] It can be understood that, is the timestamp value at the coordinate of the current timestamp marker table, which represents the timestamp value of the latest event occurring at the coordinate in the event camera coordinate system.

[0122] Step 20123. Project the a th event to the current event image to obtain a projected event image , expressed as:

[0123]

[0124] Wherein, represents the coordinate value of the projected event image a at the coordinate after the th event is projected; represents the coordinate value of the current event image at the coordinate .

[0125] Step 20124. Motion compensating the a th event to obtain a current compensated event image ;

[0126] First, obtain the coordinate a of the th eventThe expression is:

[0127] wherein, denotes the coordinate of the neighborhood point at the location; denotes the coordinate position of the a th neighborhood point of the k th event, k =1,2,…,8, wherein, denotes the a axis coordinate of the k th neighborhood point of the x th event, denotes the a axis coordinate of the k th neighborhood point of the y th event.

[0128] The neighborhood point satisfying the difference value condition in the neighborhood point coordinate set of the a th event is searched, and the motion compensation is performed on the projected event image . The specific steps are as follows:

[0129] All 8 neighborhood points of the a th event are traversed, and for the coordinate point in the neighborhood, when the difference between the coordinate value of the current timestamp marker table at the point and the coordinate value at the location is in the specified time window , wherein, denotes the minimum time difference of the same feature trigger, denotes the maximum time difference of the same feature trigger, the current coordinate value at the neighborhood point is compensated to the coordinate , and the corrected coordinate value is obtained, and the expression is:

[0130]

[0131]

[0132] wherein, denotes the coordinate value of the projected event image a at the neighborhood point GI’ after the th event is projected; denotes the corrected coordinate value of the current compensated event image at the coordinate ; denotes the corrected coordinate value of the current compensated event image at the coordinate the modified coordinate value.

[0133] obtaining a current compensated event image based on the modified coordinate value .

[0134] Step 2012. judging whether the modified coordinate value is greater than or equal to the total number of events A. a If yes, the current compensated event image is adopted; entering step 2013. a If no, setting = a +1, setting a , and returning to step 2012. a Step 2013. performing normalization processing on the current compensated event image to generate a motion deblurred event frame

[0135] , the expression of which is as follows: G

[0136]

[0137] wherein represents the coordinate value of the motion deblurred event frame in the coordinate; is the minimum coordinate value in the current compensated event image ; is the maximum coordinate value in the current compensated event image ; represents the coordinate value of the current compensated event image in the coordinate, wherein is the pixel coordinate in the x axis and the y axis.

[0138] Step 301. processing the time-aligned original frame image captured by the traditional camera module 1 to generate a down-sampled edge gradient image F .

[0139] Step 3011. performing resolution down-sampling on the original frame image FI using a bilinear interpolation method to generate a down-sampled frame picture with the same resolution as the event camera.

[0140] Specifically, a frame image with the same resolution as the event camera is determined ; for each pixel position of the down-sampled frame picture , the corresponding pixel position FI in the original frame image is calculated, and the edge gradient value of the corresponding pixel position in the original frame image is determined as the edge gradient value of the pixel position in the down-sampled frame picture.​​​ ),in, For frame images exist x shaft and y Pixel coordinates on the axis; , Original frame image exist x shaft and y The pixel coordinates on the axis are expressed as:

[0141]

[0142]

[0143] in, W f and H f These represent the pixel width and pixel height of a traditional camera, respectively. W e and H e These represent the pixel width and pixel height of the event camera, respectively.

[0144] For the original frame image FI coordinates of the points in , Get the first pixel adjacent to it. P 1( x 1, y 1) Second pixel P 2( x 2, y 1) Third pixel P 3( x 1, y 2) and the fourth pixel P 4( x 2, y 2) Perform bilinear interpolation, where, x 1 represents pixel coordinates The nearest integer on the left x Axis coordinates x 2 represents pixel coordinates The nearest integer on the right x Axis coordinates y 1 represents pixel coordinates The nearest integer on the top y Axis coordinates y 2 represents pixel coordinates The nearest integer below y Axis coordinates are used to obtain the downsampled frame image. exist coordinates at , the expression is:

[0145]

[0146]

[0147]

[0148] wherein, I 1 represents the interpolation result of the first pixel P 1 and the second pixel P 2 in the horizontal direction; I 2 represents the interpolation result of the third pixel P 3 and the fourth pixel P 4 in the horizontal direction; represents the pixel value after vertical interpolation of I 1 and I 2.

[0149] After the above steps, a down-sampled frame image with the same resolution as the event camera is obtained.

[0150] Step 3012. Obtain the edge gradient image of the down-sampled frame image F , the expression is:

[0151]

[0152]

[0153]

[0154] wherein, represents the gradient value of the down-sampled frame image in the axis direction at the pixel position x ; represents the gradient value of the image in the axis direction at the pixel position y ; represents the pixel value of the down-sampled frame image at the position , wherein, m and n are offset amounts, representing the 3×3 neighborhood pixels around the current pixel; G x ( m+ 1, n+ 1) represents the weight value of the horizontal Sobel operator; G y ( m+1, n+ 1) represents the weight value of the vertical Sobel operator; represents the down-sampled frame image at position of the edge strength.

[0155] Step 401, based on the event camera deblurred event frame G and the edge gradient image of the traditional camera F Similarity matching calculation to obtain sub-pixel level disparity d *.

[0156] Specifically, for each pixel coordinate position F in the edge gradient image of the traditional camera obtained in step 301 , in the event camera deblurred event frame obtained in step 201 G , along the disparity search range sliding window W , the normalized cross-correlation value is calculated, and the expression is:

[0157]

[0158] Wherein, NCC( d ) represents the normalized cross-correlation value under the disparity d , represents the similarity degree of the local region of the event frame G and the edge gradient image F . represents the gray value of the edge gradient image F at the coordinate , represents the gray mean value of all pixels in the sliding window F on the edge gradient image W ; represents the gray value of the event frame G at the coordinate ; represents the gray mean value of all pixels in the sliding window G on the event frame W ; d min represents the minimum candidate disparity; d max represents the maximum candidate disparity; represents the pixel offset in the x axis direction; represents the pixel offset in the y axis direction.

[0159] Further, the value of maximizing NCC is selected as the sub-pixel level disparity , and the expression is:

[0160] .

[0161] Step 501, sub-pixel level disparity is calculated pixel by pixel, and an optimized sub-pixel level disparity map is generated, and the expression is:

[0162]

[0163] wherein, represents the optimized sub-pixel level disparity value; represents the sub-pixel level disparity; , and respectively represent the NCC values at the sub-pixel level disparity center and its adjacent disparity and .

[0164] Further, when , .

[0165] The camera device and method for spatio-temporal alignment of a traditional camera and an event camera of the present application perform spatial coarse alignment through an adjustable optical device to ensure that the optical axis direction, imaging plane, and overlapping area of the view angle of the two cameras are as consistent as possible; temporal alignment is performed through a signal generator trigger, and temporal synchronization error is corrected to ensure that the two cameras collect images or events at the same time; through the generation of an event frame eliminating motion blur, sub-pixel level disparity optimization, high-precision spatial alignment of the event camera and the traditional camera at the pixel level is achieved. The data of the traditional camera and the event camera after spatio-temporal alignment can be used for modal fusion, such as robot navigation, target detection, three-dimensional reconstruction, and environmental perception for subsequent tasks.

[0166] The above merely describes a preferred embodiment of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for spatio-temporal alignment of a traditional camera and an event camera, characterized in that, A space-time alignment method for a space-time alignment camera for traditional camera and event camera space-time alignment, the camera comprising an adjustable optical device, a device base platform, a computer and a signal generator; The adjustable optical device comprises a traditional camera module, an event camera module, a light splitting module and a device base platform; The computer and the signal generator are connected with the traditional camera module and the event camera module; The traditional camera module comprises a traditional camera; the traditional camera is movably arranged on the device base platform in height; The event camera module comprises an event camera; the event camera is movably arranged on the device base platform in height; The traditional camera and the event camera are arranged at a 90-degree angle; The light splitting module comprises a light splitting mirror, which is arranged between the traditional camera module and the event camera module; The specific steps of the space-time alignment method are as follows: Step 1. Obtain the event stream of the event camera and the original frame image of the traditional camera; Step 2. Obtain the trigger timestamp data of the traditional camera and the event camera; Step 3. Correcting threshold determined according to trigger frequency Trigger timestamp interval of event camera Smoothly correcting trigger timestamp of event camera, obtaining corrected trigger timestamp of event camera Step 4. Establish the trigger timestamp mapping relationship of the traditional camera and the event camera, map the corrected trigger timestamp of the event camera into the trigger timestamp reference system of the traditional camera, and obtain the trigger timestamp of the mapped event camera; Step 5. Based on the trigger timestamp of the mapped event camera, obtain the time synchronization error of the traditional camera and the event camera; Step 6. Based on the time synchronization error of the traditional camera and the event camera, obtain the mean deviation of the synchronization error; Step 7. Align the timestamp of the traditional camera using the mean deviation of the synchronization error to obtain the aligned timestamp of the traditional camera; The determination condition for smoothing correction of the trigger timestamp of the event camera according to the correction threshold determined according to the trigger frequency and the trigger timestamp interval of the event camera is: if The first event camera i Trigger timestamp The trigger timestamp has been corrected as the data was triggered erroneously. if and The first event camera i Trigger timestamp To ensure effective data triggering, the event camera trigger timestamp is used as the event camera correction trigger timestamp; Indicates the time interval between two trigger signals; If and , the first trigger timestamp of the event camera i is false trigger data, correct the timestamp. ​ 2. The spatio-temporal alignment method of claim 1, wherein, The light splitting module further comprises a light shielding main body, a first lens cover and a second lens cover; the light splitting mirror is arranged inside the light shielding main body; the first lens cover and the second lens cover are arranged on two adjacent outer sides of the light shielding main body respectively; the first lens cover is arranged opposite to the traditional camera module, and the second lens cover is arranged opposite to the event camera module.

3. The spatio-temporal alignment method of claim 1, wherein, The optical axes of the traditional camera and the event camera are in the same plane and perpendicular to each other.

4. The spatio-temporal alignment method of claim 1, wherein, The light splitting mirror is arranged at the intersection of the optical axes of the traditional camera and the event camera.

5. The spatio-temporal alignment method of claim 1, wherein, The expression of the trigger timestamp of the mapped event camera is: wherein, represents the mapped event camera's first i trigger timestamp; represents the clock rate scale factor; represents the clock offset; represents the event camera's first i trigger timestamp 's correction value; N represents the real number of trigger signals.

6. The spatio-temporal alignment method of claim 5, wherein, Clock rate scaling factor The expression for the clock rate scaling factor is: wherein, represents the first trigger timestamp of the conventional camera. j represents the second trigger timestamp of the conventional camera.

7. The spatio-temporal alignment method of claim 6, wherein, Clock offset The expression for the clock offset is: wherein, represents the first j trigger timestamp of the conventional camera.

8. A method for spatial alignment of a traditional camera and an event camera, the method comprising: The specific steps are as follows: the space-time alignment method for the traditional camera and the event camera according to any one of claims 1-7 obtains the time-aligned event stream and the original frame image, and performs spatial alignment of the traditional camera and the event camera based on the original frame image and the event stream.

Images