Method for obtaining high dynamic image ground truth at any time instant of an event camera

Abstract

This invention discloses a method for obtaining the ground truth of high dynamic range images at any time from an event camera, comprising the following steps: calibrating multiple optical cameras to obtain the homography matrix H between each optical camera and a specified optical camera. k Then, images captured by two optical cameras at different exposure values ​​are obtained in the dynamic scene, and these images are compared with the homography matrix H. k Weighted image fusion is performed to obtain a high dynamic range (HDR) image. Using SIFT feature extraction and traversal matching, key points in adjacent frames of the HDR image are matched. If the number of matching points is sufficient, the homography matrix H between the two adjacent frames is calculated; if the number is insufficient, the threshold is reduced, and matching is performed again. The homography matrix H is linearly interpolated and multiplied with the previous frame of the two adjacent frames to obtain the HDR image at any given time. This invention effectively solves the problem of obtaining the true value of an HDR image at any given time.

Description

Technical Field

[0001] This invention belongs to the field of image processing technology, and more specifically, relates to a method for obtaining the ground truth of a high dynamic range image at any time from an event camera. Background Technology

[0002] HDR (High Dynamic Range Imaging) is a set of techniques used to achieve a greater dynamic range of events than ordinary digital images. One common method is to select appropriate portions from a set of images with different exposure ranges and fuse them to remove overexposed or underexposed pixels, thus obtaining a high dynamic range image. An event camera is a new type of dynamic sensor with advantages such as low latency, low power consumption, high temporal resolution, and high dynamic range. It can asynchronously detect the logarithmic change in illumination for each pixel, and when the change exceeds a certain threshold, it asynchronously outputs event information including pixel coordinates, timestamps, and polarity. Compared to traditional cameras that output images frame by frame, event camera pixels respond independently to dynamic changes without exposure time constraints, resulting in an extremely large dynamic range and great potential for application in HDR image reconstruction.

[0003] Research on HDR fusion images from event cameras is increasing, and the ground truth of HDR images is a necessary factor for evaluating the quality of HDR images from event cameras. Because of the working principle of event cameras, they can output data at any time, without being limited by the frame rate. Therefore, algorithms related to event cameras require the ground truth of the image at any time, i.e., the ground truth of the image unconstrained by the frame rate. However, due to limitations in equipment and technology, it is difficult to obtain the ground truth of the HDR image at any time from an event camera. Currently, patent CN114708478A discloses a "method, apparatus, device, and medium for data fusion between event cameras and standard cameras," which fuses event camera data and standard camera data using point cloud block information; patent CN114862732A discloses a "method for synthetic aperture imaging by fusing event cameras and traditional optical cameras," which reconstructs high-quality, unobstructed images by synthesizing the aperture of event cameras and traditional optical cameras; and patent CN114463399A discloses a "spatiotemporal matching method between event cameras and traditional optical cameras," which achieves spatiotemporal matching between event cameras and traditional cameras by performing matching searches using scene structure similarity indicators. However, the above addresses issues such as imaging and registration between an event camera and a traditional camera, and cannot solve the difficulty of obtaining the true value of an event camera's HDR image at any given moment. Summary of the Invention

[0004] To address the shortcomings of existing technologies, the present invention aims to provide a method for acquiring high dynamic range images at any given time using multiple optical cameras, thereby solving the problem that current HDR image algorithms involving event cameras cannot obtain the true value of HDR images at any given time.

[0005] To achieve the above objectives, the present invention provides a method for obtaining the ground truth value of a high dynamic range image at any time from an event camera, comprising the following steps:

[0006] (1) Calibrate multiple optical cameras whose optical centers are equivalent to the same position to obtain the homography matrix H between each optical camera and a specified optical camera. k Then, in a dynamic scene, a series of images with the same timestamp are obtained, captured by multiple optical cameras at different exposure values. These images are then compared with the homography matrix H. k Image fusion is performed using weighted methods to obtain high dynamic range images;

[0007] (2) By using SIFT feature extraction and traversal matching, the feature information of key points in all frames of the high dynamic range image is determined, and the key points in two adjacent frames of the high dynamic range image are matched. If the number of matched key points is within the number threshold, then step (3) is executed; if the number of matched key points is not within the number threshold, then the number threshold is reduced, and step (2) is executed again; wherein, the feature information of the key points includes the position, scale and main direction information of the key points;

[0008] (3) Based on the location information of the key points matched in two adjacent frames, calculate the homography matrix H between the two adjacent frames, and perform linear interpolation on the homography matrix H to obtain the homography matrix H′ at any time. Then, judge the rationality of the calculation of the homography matrix H′. If the calculation is reasonable, combine the homography matrix H′ with the previous frame in the two adjacent frames to obtain the high dynamic range image at any time. If the calculation is unreasonable, calculate and judge the homography matrix H between the next pair of adjacent frames.

[0009] This invention provides a method for obtaining the ground truth of high dynamic range images at any time from an event camera. It acquires images from multiple traditional optical cameras at the same time, uses a homography matrix and image fusion method to obtain high dynamic range images, and then obtains the homography matrix between images based on SIFT feature matching. By linear interpolation, the high dynamic range image at any time can be obtained, which can effectively solve the problem of obtaining the ground truth of related images from an event camera.

[0010] In one embodiment, in step (1), the high dynamic range image R obtained by image fusion is... ij Represented as:

[0011]

[0012]

[0013]

[0014]

[0015]

[0016]

[0017] In the formula, W ij,k The weights represent the weights used in the weighted fusion process described in image fusion; k represents the optical camera index; N represents the total number of optical cameras; I ij,k σ represents the pixel at coordinates (i,j) in the image transformed from the k-th optical camera, i.e., the pixel value of the image from all optical cameras transformed to the viewpoint of a specified optical camera using a homography matrix; k This represents the scale parameter of the image converted from the k-th optical camera; This represents the normalized weights after mapping all image weights; and The corresponding values ​​represent the brightness of the blue, green, and red channels of pixel (i,j) in the image converted from the k-th optical camera; C ij,k S ij,k and E ij,k The corresponding values ​​represent the contrast, saturation, and brightness of pixel (i,j) in the image converted from the k-th optical camera, ω. C ω S and ω E The corresponding values ​​represent the weights of contrast, saturation, and brightness.

[0018] In one embodiment, step (2), which involves determining the key point feature information of all frames in the high dynamic range image using the SIFT feature extraction and traversal matching method, and performing feature matching on the key points in two adjacent frames of the high dynamic range image, specifically includes:

[0019] (2.1) Downsample and Gaussian blur at different scales for each frame of the high dynamic range image to establish a Gaussian pyramid;

[0020] (2.2) Gaussian difference pyramids are obtained by subtracting the convolution of Gaussian kernels of different scales. Then, by finding local extreme points, the position and scale of all key points in each frame image are determined. The gradient magnitude and direction of the key points are calculated according to the Gaussian pyramids to determine the main direction of the key point.

[0021] (2.3) The position, scale and main direction of each key point in each frame image are described to obtain the descriptor vector of the key point. Then, the similarity matching degree of each key point in two adjacent frames of the high dynamic range image is calculated based on the descriptor vector. If the similarity matching degree exceeds the matching threshold, the key point is identified as a matching point.

[0022] In one embodiment, in step (3), the homography matrix H between two adjacent frames is:

[0023]

[0024] In the formula, u and v represent the x and y coordinates of the key points being matched. This represents the x-coordinate of the Mth matching keypoint in the previous frame of two adjacent frames. h1 represents the x-coordinate of the Mth matching key point in the next frame of two adjacent frames, where the four matching key points are not collinear; h1 to h8 represent the eight parameters in the homography matrix H.

[0025] In one embodiment, in step (2.2), the formulas for calculating the gradient magnitude m(x,y) and direction θ(x,y) of the key point are as follows:

[0026]

[0027]

[0028] In the formula, L represents the scale space function of the Gaussian pyramid; (x,y) represents the pixel position of the image in the Gaussian pyramid.

[0029] In one embodiment, step (2.2), the step of calculating the gradient magnitude and direction of the key point and determining the main direction of the key point, specifically includes:

[0030] Using the keypoint as the center, the gradient direction and magnitude of all pixels within a circle with a radius of 1.5 times the Gaussian image scale where the keypoint is located are determined by a Gaussian weighted average and voting mechanism to determine the main direction of the keypoint.

[0031] In one embodiment, step (2.3), which involves performing feature description on the position, scale, and principal direction of each key point in each frame image to obtain the descriptor vector of the key point, specifically includes:

[0032] Rotate the coordinate axis to the main direction of the keypoint, divide its neighborhood into several regions of the same size, calculate the gradient magnitude and direction of all pixels in the region, and apply Gaussian weighted summation and voting mechanism to the magnitude and direction in each neighborhood to obtain the normalized direction and magnitude of each region, which is the descriptor vector of the keypoint.

[0033] In one embodiment, the similarity matching degree d(R) i ,S i The formula for calculating ) is:

[0034]

[0035] In the formula, r ij s ij These correspond to the key point descriptor vectors in the previous and next frames of two adjacent images, respectively.

[0036] In one embodiment, in step (3), the formula for calculating the homography matrix H′ is:

[0037]

[0038] In the formula, t m t0 represents the timestamp of any moment within the time interval between the capture of two adjacent frames; t1 represents the timestamp of the capture of the previous frame in the two adjacent frames; t2 represents the timestamp of the capture of the next frame in the two adjacent frames.

[0039] In one embodiment, in step (3), the basis for judging the rationality of the calculation of the homography matrix H′ is: multiply the homography matrix H′ with the previous frame image in the two adjacent frames, and calculate the average value of the sum of squared distances of each key point in the resulting image and the next frame image in the two adjacent frames. If the average value is greater than the set value, the homography matrix H′ is judged to be calculated reasonably; otherwise, the homography matrix H′ is judged to be calculated unreasonably. Attached Figure Description

[0040] Figure 1 This is a flowchart of the method for obtaining the true value of a high dynamic range image at any moment from an event camera, provided by the present invention.

[0041] Figure 2 This is a flowchart illustrating the overall implementation of the algorithm for the method of obtaining the ground truth value of a high dynamic range image at any moment from an event camera, as provided in this invention.

[0042] Figure 3 This is a schematic diagram of the acquisition control circuit provided in an embodiment of the present invention;

[0043] Figure 4 This is a schematic diagram of the result of obtaining an HDR image according to a specific embodiment of the present invention;

[0044] Figure 5 This is a schematic diagram of the hardware platform structure provided in a specific embodiment of the present invention. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention.

[0046] This invention provides a method for obtaining the true / reference value of a high dynamic range image at any time from an event camera, such as... Figure 1 As shown, the method includes steps S10 to S30, which are detailed below:

[0047] S10, calibrate multiple conventional optical cameras whose optical centers are equivalent to the same location, and obtain the homography matrix H between each optical camera and one of the specified optical cameras. k Then, in a dynamic scene, a series of images with the same timestamp are obtained, captured by multiple optical cameras at different exposure values; these images are then compared with the homography matrix H. k Image fusion is performed using weighted methods to obtain a high dynamic range image.

[0048] In step S10, camera calibration can be performed using traditional camera calibration methods, such as the checkerboard calibration method, to obtain the homography matrix H between each optical camera and a specified optical camera. k The expression is as follows:

[0049]

[0050] Homography matrix H k h represents the homography matrix between the k-th optical camera and one of the specified optical cameras. 11,k ~h 32,k H represents the homography matrix k The eight parameters in it. It should be noted that the homography matrix H... k H2 is a 3x3 identity matrix. The third row and third column of the homography matrix between all optical cameras and one designated optical camera are both 1. Assuming the number of the designated optical camera is 2, then the homography matrix H2 is a 3x3 identity matrix.

[0051] Then the image from the k-th camera converted to the image from one of the specified optical cameras is H. k I k .

[0052] In image fusion, the weights W described in the weighted fusion are... ij,k The calculation formula is:

[0053]

[0054] In the formula, C, S, and E represent contrast, saturation, and brightness, respectively. ij,k S ij,k and E ij,kThe corresponding values ​​represent the contrast, saturation, and brightness of the pixel at coordinates (i,j) in the k-th image; ω C ω S and ω E The corresponding values ​​represent the weights of contrast, saturation, and brightness.

[0055] Specifically, the formulas for calculating contrast, saturation, and brightness are as follows:

[0056]

[0057]

[0058]

[0059]

[0060] In the formula, and The corresponding values ​​represent the brightness of the blue, green, and red channels of pixel (i,j) in the image converted from the k-th optical camera; I ij,k σ represents the pixel at coordinates (i,j) in the image transformed from the k-th optical camera, i.e., the pixel value of the image from all optical cameras transformed to the viewpoint of a specified optical camera using a homography matrix; k This represents the scale parameter of the image converted from the k-th optical camera.

[0061] In image fusion, the weights are normalized. The normalization formula is as follows:

[0062]

[0063] High dynamic range image R obtained in image fusion ij for:

[0064]

[0065] Step S20 involves using SIFT feature extraction and traversal matching to determine the feature information of key points in all frames of the high dynamic range image. Key points in adjacent frames of the high dynamic range image are then matched. If the number of matched key points is within a threshold, the subsequent step S30 is executed; otherwise, the threshold is lowered, and step S20 is repeated. The feature information of the key points includes their position, scale, and main orientation.

[0066] Specifically, step S20 is implemented as follows: Each frame of the high dynamic range image is downsampled and Gaussian blurred at different scales to establish a Gaussian pyramid; then, a Gaussian difference pyramid is obtained by subtracting the convolution of Gaussian kernels at different scales; furthermore, by finding local extrema, the position and scale of all key points in each frame are determined; then, the gradient magnitude and direction of the key points are calculated to determine the principal direction of the key point; finally, the position, scale, and principal direction of each key point in each frame are described to obtain the descriptor vector of the key point; then, the similarity matching degree of each key point in two adjacent frames of the high dynamic range image is calculated based on the descriptor vector; if the similarity matching degree exceeds the matching threshold, the key point is identified as a matching point; if the number of matching points is within the number threshold, the subsequent step S30 is executed; otherwise, the number threshold is reduced, and step S20 is executed again.

[0067] In this embodiment, the scale space function L(x,y,σ) in the Gaussian pyramid is:

[0068] L(x,y,σ)=G(x,y,σ)*R(x,y)

[0069]

[0070] In the formula, G(x,y,σ) represents the Gaussian function; * represents convolution calculation; a,b represent the dimensions of the Gaussian template; (x,y) represents the pixel position of the image in the Gaussian pyramid; and σ represents the scale parameter of the image in the Gaussian pyramid, i.e., the blur coefficient.

[0071] The gradient magnitude and direction at key points are calculated as follows:

[0072]

[0073]

[0074] The Gaussian difference pyramid D(x,y,σ) represents the difference between two adjacent levels within the same level of the Gaussian pyramid. The formula for its calculation is:

[0075] D(x,y,σ)=[G(x,y,σ f )-G(x,y,σ f-1 )]=L(x,y,σ f )-L(x,y,σ f-1 )

[0076] In the formula, σ f denoted as the fuzziness coefficient of the f-th level Gaussian pyramid.

[0077] Furthermore, in feature extraction and matching, the location and scale of extreme points, i.e. key points, can be found by comparing the magnitudes of the Gaussian differences of surrounding pixels and pixels at different levels.

[0078] Where m(x,y) represents the gradient magnitude of the keypoint, and θ(x,y) represents the gradient direction of the keypoint.

[0079] Furthermore, the gradient direction and magnitude of all pixels within a circle centered on the keypoint and with a radius 1.5 times the scale of the Gaussian image containing the keypoint are statistically analyzed, and the main direction of the keypoint is determined through a Gaussian weighted average and voting mechanism.

[0080] Furthermore, the location, scale, and orientation of keypoints are used to describe features. First, the coordinate axes are rotated to the main direction of the keypoint, and its neighborhood is divided into several regions of the same size. The gradient magnitude and direction of all pixels in the region are calculated. The magnitude and direction in each neighborhood are Gaussian weighted and voted to obtain the normalized direction and magnitude of each region, which is the descriptor vector of the keypoint.

[0081] Furthermore, based on the descriptor vectors, the formula for calculating the similarity matching degree of each keypoint in two adjacent frames of a high dynamic range image is as follows:

[0082]

[0083] In the formula, R i S i R represents the key point descriptors in the previous and next frame images. i =(r i1 ,r i2 ,…,r iN ), S i =(s i1 ,s i2 ,…,s iN Key points whose similarity matches exceed the matching threshold can be identified as matching points.

[0084] S30. Based on the position information of the matched key points in two adjacent frames, calculate the homography matrix H between the two adjacent frames, and perform linear interpolation on the homography matrix H to obtain the homography matrix H′ at any time. Then, determine the rationality of the calculation of the homography matrix H′. If the calculation is reasonable, multiply the homography matrix H′ with the previous frame in the two adjacent frames to obtain the high dynamic range image at any time. If the calculation is unreasonable, perform the calculation and judgment of the homography matrix H between the next pair of adjacent frames.

[0085] In step S30, in step (3), the homography matrix H between two adjacent frames is:

[0086]

[0087] In the formula, u and v represent the x and y coordinates of the key points being matched. This represents the x-coordinate of the Mth matching keypoint in the previous frame of two adjacent frames. h1 represents the x-coordinate of the Mth matching key point in the next frame of two adjacent frames, where the four matching key points are not collinear; h1 to h8 represent the eight parameters in the homography matrix H.

[0088] The formula for calculating the homography matrix H′ at any given time is as follows: (This is a linear interpolation of the homography matrix H.)

[0089]

[0090]

[0091] In the formula, t m t0 represents the timestamp of any moment between two frames; t0 represents the timestamp of the previous frame; t1 represents the timestamp of the next frame.

[0092] Furthermore, the criterion for judging the reasonableness of the homography matrix H′ provided in this embodiment can be as follows: multiply the homography matrix H′ with the previous frame image in the two adjacent frames to obtain the average value of the sum of squared distances between the key points of the resulting image and the subsequent frame image. If it is greater than a set value, the calculation is unreasonable, which is expressed as follows:

[0093] I″=H′I p

[0094]

[0095] In the formula, I p I represents the previous frame in two adjacent frames. n The next frame in a pair of adjacent frames; x I″ y I″ The corresponding representation is the x and y coordinates of the key points in the image obtained by multiplying the homography matrix H′ by the previous frame image in the two adjacent frames; These correspond to the x and y coordinates of key points in the next frame of the two adjacent image frames. The reasonableness setting value is 1.

[0096] If the calculation of the homography matrix H′ is deemed reasonable, then the homography matrix H′ is compared with the previous frame image I in the two adjacent frames. p Multiply to obtain the high dynamic range image I at any time. tm for:

[0097] I tm =H′I p

[0098] The method for obtaining the ground truth of high dynamic range images at any time provided in this embodiment acquires images from multiple traditional optical cameras at the same time, uses a homography matrix and image fusion method to obtain high dynamic range images, and then obtains the homography matrix between images based on SIFT feature matching. By linear interpolation, the high dynamic range image at any time can be obtained, which can effectively solve the problem of obtaining the ground truth of related images of event cameras.

[0099] On the other hand, this application provides a synchronous image acquisition control circuit, such as... Figure 3 As shown, the circuit includes a power supply module, a PWM module, a CMOS module, and a camera module.

[0100] The power supply module supplies power to the camera module and CMOS module, providing the stable voltage required by each component; the PWM module outputs a PWM signal to synchronize the shooting time of the camera module; the CMOS module boosts the PWM signal to adapt to the voltage required by the subsequent camera module; and the camera module receives circuit signals and synchronously acquires image data.

[0101] Specifically, the power supply module is connected to the CMOS and voltage regulator modules to ensure appropriate voltage supply to each module. The PWM module is connected to the voltage regulator module to obtain a stable voltage from the power supply module after passing through the voltage regulator, and outputs a stable square wave signal. The CMOS module is connected to the PWM module, receives the PWM output signal, and inverts and boosts it. The camera module consists of multiple identical optical cameras and one event camera. The PWM signal is directly connected to the event camera, and the ordinary cameras receive the signal inverted and boosted by the CMOS, simultaneously capturing data upon receiving the signal. The event camera software is configured to generate a synchronization event on the rising edge. The software for the two optical cameras is configured to capture images on the falling edge.

[0102] Furthermore, this invention also provides a hardware platform, which includes a carrier plate, a camera support module, and a beam splitter module. The carrier plate supports all components and ensures they are horizontal; the camera support module fixes the camera and ensures it is on the same horizontal plane; the beam splitter module separates the input light rays, ensuring that each lens captures the same image.

[0103] Specifically, the carrier plate is a thin cuboid platform with several threaded holes for fixing the camera support module and the beam splitter module. The camera support module uses an "L"-shaped bracket with several through holes on multiple sides. The bottom surface mates with the threaded holes on the carrier plate, and the sides mate with the threaded holes on the camera. The beam splitter module consists of a cuboid base plate, multiple side plates, and an end cap. The cuboid base plate has two parallel grooves for fixing the beam splitter lens. The side plates are hollowed out in the middle for the lens to enter, and have through holes at the bottom for mates with the threaded holes on the cuboid base plate. The end cap has two parallel grooves aligned with the base plate for fixing the beam splitter lens.

[0104] The solution provided by the present invention will be described in detail below with reference to specific embodiments.

[0105] like Figure 5 As shown, the hardware platform provided in this specific embodiment includes: a conventional optical camera, an event camera, a beam splitter, and a synchronization circuit. In the beam splitter module, two beam splitters are arranged parallel to the edge of the base plate at a 45° angle. Multiple conventional optical cameras are connected through the beam splitters and use lenses of the same specifications. Both conventional optical cameras are selected as HIKROBOTMV-CA013-21UC models. The design specifies that the exposure time of one camera is four times that of the other, used to acquire image frame data with different exposure ranges and transmit it to the computer terminal. The event camera is selected as a DAVIS346.

[0106] The overall flowchart of this embodiment is as follows: Figure 2 As shown, the specific steps are as follows:

[0107] Step 1: First, use a checkerboard pattern to calibrate multiple cameras and obtain the homography matrix of multiple cameras. Then, through the control circuit, obtain a series of image frames acquired with the same timestamp in a dynamic scene. Then, use the homography matrix obtained above and the image weighted fusion to obtain a high dynamic range image.

[0108] The weighting function described in step 1 is:

[0109]

[0110] Among them, C ij,k S ij,k and E ij,k The corresponding values ​​represent the contrast, saturation, and brightness of pixel (i,j) in the image converted from the k-th optical camera, ω. C ω S and ω E To assign a weight, we typically set ω. C =ω S =ω E =1.

[0111] Step 2: First, downsample the original image and apply Gaussian blur at different scales to build a Gaussian pyramid. Then, convolve and subtract the Gaussian pyramids using Gaussian kernels of different scales to obtain a Gaussian difference pyramid. Next, determine the location of key points by finding local extrema. Then, calculate the magnitude and direction of the gradient to determine the main direction of the key points. Finally, perform feature description based on the location, scale, and direction of the key points, calculate the similarity matching degree, and complete the key point matching. If the number of key points is sufficient, proceed to step 3. If not, halve the number threshold and repeat step 2.

[0112] The detailed process of feature description of key points in step 2 is as follows: Figure 3 First, the image region required for the descriptor is calculated and the principal direction is corrected. Generally, a 4×4×8=128-dimensional vector is selected for representation. To ensure rotation invariance, the x-axis of the original image is rotated to the same direction as the principal direction. Then, the gradient magnitude and gradient direction of each pixel in the neighborhood are calculated and weighted using a Gaussian window. Finally, a gradient histogram with 8 directions is drawn on each small block, and the cumulative value of each gradient direction is calculated to form a seed point. Each keypoint consists of 4 seed points, and each seed point has vector information in 8 directions.

[0113] In step 2, to enhance the robustness of the matching, each keypoint is typically described using 16 seed points (4×4). This allows each keypoint to generate a 128-dimensional feature vector with an extremely low repetition probability. The keypoint number threshold in step 2 is set to 80. In step 2, the keypoint number threshold is then reduced to 80% of the previous threshold.

[0114] The final similarity measure in step 2 is expressed as follows:

[0115]

[0116] Wherein d(R) i ,S i ) is the similarity evaluation index between two consecutive frames of images, r ij s ij The corresponding key point descriptors represent the key points of the preceding and following frames, with N being 128.

[0117] In step 2, the similarity parameter threshold is 1.

[0118] Step 3: As Figure 4 As shown, feature matching between two consecutive frames is achieved in step 2 to obtain the homography matrix of the two consecutive frames. Then, the homography matrix H′ at any time is obtained by linear interpolation. If the homography matrix H′ is calculated reasonably, it can be multiplied with the previous frame image to obtain the high dynamic range image at any time. If it is not reasonable, the next pair of adjacent frames is processed directly.

[0119] The linear interpolation described in step 3 is expressed as:

[0120]

[0121] In the formula, H′ is the homography matrix between any time and the time of the previous frame, H is the homography matrix between the two frames, and t m The timestamps are any points within the time interval between the capture of two frames, where t0 is the timestamp of the previous frame and t1 is the timestamp of the next frame.

[0122] The criteria for judging the reasonableness of the homography matrix calculation in step 3 are as follows:

[0123] I″=H′I p

[0124]

[0125] In the formula, I p For the previous frame in two adjacent frames, I n This refers to the next frame in a pair of adjacent frames. The reasonableness setting is 1.

[0126] The circuit schematic of this invention is as follows: Figure 3 As shown, where:

[0127] The power supply voltage is 12V. The voltage regulator module converts 12V to 3.3V. The PWM output is a square wave with an amplitude of 3.3V. After passing through the CMOS module, the voltage is boosted to 12V square wave, which in turn controls multiple ordinary cameras to acquire signals on the same falling edge. The event camera directly receives the square wave signal with an amplitude of 3.3V output by the PWM module and acquires the signal on the rising edge.

[0128] Those skilled in the art will readily understand that the above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for obtaining ground truth of high dynamic range image at any time instant of an event camera, characterized in that, The method comprises the following steps: (1) calibrate multiple optical cameras equivalent to the same position of the optical center, get the homography matrix H between each optical camera and a designated optical camera k Then a series of images with the same timestamp are obtained by multiple optical cameras under different exposure values in a dynamic scene, and the image and homography matrix H k Weighted image fusion to get high dynamic range image; (2) determining the feature information of the key points in all frame images in the high dynamic range image by a SIFT feature extraction and traversal matching method, and performing feature matching on the key points in two adjacent frame images in the high dynamic range image, if the number of matched key points is within a number threshold, then step (3) is performed, if the number of matched key points is not within the number threshold, then the number threshold is reduced, and step (2) is re-performed; wherein the feature information of the key points comprises position, scale and main direction information of the key points; (3) calculating a homography matrix H between two adjacent frame images according to the position information of the matched key points in the two adjacent frame images, performing linear interpolation on the homography matrix H to obtain a homography matrix H' at any time, and then judging the calculation rationality of the homography matrix H', if the calculation is rational, then combining the homography matrix H' with a previous frame image in the two adjacent frame images to obtain a high dynamic range image at any time, if the calculation is not rational, then performing the calculation and judgment of the homography matrix H between the next pair of adjacent frame images.

2. The method of claim 1, wherein, In step (1), the high dynamic range image R obtained by image fusion ij is represented as: where W ij,k denotes the weight of the weighted fusion described in image fusion; k denotes the optical camera serial number; N denotes the total number of optical cameras; I ij,k denotes the pixel of the coordinate (i, j) in the transformed image of the kth optical camera, i.e. the pixel value of the image of all optical cameras converted to the image under the specified optical camera view through the homography matrix respectively; σ k denotes the scale parameter of the transformed image of the kth optical camera; denotes the weight of the weight mapping of all images after normalization; and correspond to the blue, green and red channel brightness of the pixel of the coordinate (i, j) in the transformed image of the kth optical camera; C ij,k , S ij,k and E ij,k correspond to the contrast, saturation and brightness of the pixel of the coordinate (i, j) in the transformed image of the kth optical camera, ω C , ω S and ω E correspond to the weights of the contrast, saturation and brightness.

3. The method of claim 1, wherein, In step (2), the step of determining the feature information of the key points in all frame images in the high dynamic range image by the SIFT feature extraction and traversal matching method, and performing feature matching on the key points in two adjacent frame images in the high dynamic range image, is specifically: (2.1) performing down-sampling and Gaussian blur of different scales on each frame image in the high dynamic range image to establish a Gaussian pyramid; (2.2) obtaining a Gaussian difference pyramid by using different scale Gaussian kernels and their convolution subtraction, and then determining the position and scale of all key points in each frame image by finding local extreme points, and determining the gradient amplitude and direction of the key points according to the Gaussian pyramid to determine the main direction of the key points; (2.3) performing feature description on the position, scale and main direction of each key point in each frame image to obtain a descriptor vector of the key points, and then calculating the similarity matching degree of each key point in two adjacent frame images in the high dynamic range image according to the descriptor vector, if the similarity matching degree exceeds a matching threshold, then the key point is determined as a matched point.

4. The method of claim 3, wherein, In step (3), the homography matrix H between two adjacent frame images is: wherein u, v respectively represent the horizontal and vertical coordinates of the matched key point, represents the horizontal coordinate of the Mth matched key point in the previous one of the two adjacent image frames, represents the horizontal coordinate of the Mth matched key point in the latter one of the two adjacent image frames, wherein the four matched key points are not collinear; h1-h8 represent eight parameters in the homography matrix H.

5. The method of claim 3, wherein the true value of the high dynamic range image at any time of the event camera is obtained by: In step (2.2), the calculation formulae of the gradient amplitude m(x, y) and direction θ(x, y) of the key points are: In the formula, L represents a scale space function of the Gaussian pyramid; (x, y) represents the pixel position of the image in the Gaussian pyramid.

6. The method of claim 3 or 5, wherein the true value of the high dynamic range image at any time of the event camera is obtained. In step (2.2), the step of calculating the gradient amplitude and direction of the key points to determine the main direction of the key points is specifically: The gradient direction and amplitude of all pixels within a circle with a radius of 1.5 times the scale of the Gaussian image where the key point is located and the key point as the center are determined by a Gaussian weighting and voting mechanism to determine the main direction of the key point.

7. The method of claim 3, wherein the true value of the high dynamic range image at any time of the event camera is obtained by: In step (2.3), the step of performing feature description on the position, scale and main direction of each key point in each frame image to obtain a descriptor vector of the key points is specifically: The coordinate axis is rotated to the main direction of the key point, the neighborhood of the key point is divided into several regions of the same size, the gradient amplitude and direction of all pixel points in the regions are calculated, the amplitude and direction of each region are weighted by Gauss and the voting mechanism, the normalized direction and amplitude of each region are obtained, and the normalized direction and amplitude are the descriptor vector of the key point.

8. The method of claim 3, wherein, The similarity matching degree d(R i ,S i ) is calculated by the following formula: In the formula, r ij , s ij Corresponding respectively represent the key point descriptor vectors in the previous frame image and the next frame image in the adjacent two frame images.

9. The method of claim 1, wherein, In step (3), the calculation formula of the homography H' is: In the formula, t m denotes a timestamp at any moment within the time interval between the photographing of the two adjacent frames of images; t0 represents a time stamp of a previous frame image in the two adjacent frames of images; t1 represents a time stamp of a later frame image in the two adjacent frames of images.

10. The method of claim 1, wherein, In step (3), the judgment basis for the calculation rationality of the homography H' is that the homography H' is multiplied by the previous frame image in the two adjacent frames of images, and the average value of the distance squares of the key points in the obtained image and the later frame image in the two adjacent frames of images is calculated, if the average value is greater than a set value, it is judged that the homography H' is calculated reasonably, otherwise, it is judged that the homography H' is calculated unreasonably.

Images