An event frame image alignment method based on space-time gradient propagation

Abstract

The application relates to a kind of event frame image alignment methods based on space-time gradient propagation, belong to image processing and computer vision technical field, solve the problem of existing technology in the dependence of camera pose estimation, only in a few specific shooting scene application.The alignment method of the application has no shooting scene limit, by sequentially processing event, establishing history matrix and cumulative processing, using the gradient information of event in local space-time neighborhood for motion fitting and coordinate correction, to realize accurate alignment of sub-pixel level.The application not only effectively enhances the accuracy of event information, but also has good effect on noise suppression, can well adapt to dynamic scene event data processing.

Description

Technical Field

[0001] The present invention relates to the technical field of image processing and computer vision, and in particular to an event frame image alignment method based on spatiotemporal gradient propagation. Background Art

[0002] With the development of event camera technology, capturing asynchronous events through pixel-level brightness changes has become an important means of solving problems in high-speed dynamic scenes. Traditional image processing methods mainly rely on continuous frame images, which often cannot meet the requirements of high spatiotemporal resolution for fast-moving or changing scenes. When using event data for image processing, the asynchronous event data needs to be preprocessed, which often requires de-motion blurring through alignment. Existing event alignment methods are usually based on coordinate reprojection based on camera pose estimation. This often makes it difficult to accurately capture local motion information in environments where the camera is fixed and there are moving objects in the scene, resulting in poor alignment results or even outright failure.

[0003] To address the scene limitations of traditional alignment methods, this paper proposes an event frame image alignment method based on spatiotemporal gradient propagation. This method is scene-independent and achieves sub-pixel precision alignment by sequentially processing events, establishing a history matrix, and performing cumulative processing. It leverages the gradient information of events within their local spatiotemporal neighborhood for motion fitting and coordinate correction. This technical solution not only effectively enhances the accuracy of event information but also effectively suppresses noise, making it well-suited for event data processing in dynamic scenarios. Summary of the Invention

[0004] In view of the above problems, the present invention provides an event frame image alignment method based on spatiotemporal gradient propagation, which solves the problem in the prior art that it relies on camera pose estimation and can only be applied in a few specific shooting scenarios.

[0005] The present invention provides an event frame image alignment method based on spatiotemporal gradient propagation, the specific steps of which are as follows:

[0006] Step S1: Divide the event stream data captured by the event camera into multiple event stream segments;

[0007] Step S2: construct a history matrix and a cumulative matrix of the same size as the image resolution;

[0008] Step S3: Process the events in the event stream segment one by one, obtain the spatiotemporal gradient of the current event based on the historical events in the spatiotemporal neighborhood of the history matrix; obtain the fitting starting point of the current event based on the spatiotemporal gradient of the current event and save it in the history matrix;

[0009] Step S4: Calculate the corrected coordinates of the corresponding event based on the spatiotemporal coordinates of each event, the spatiotemporal coordinates of the fitting starting point, and the alignment time to obtain an updated cumulative matrix;

[0010] Step S5: After all events in the event stream segment are processed according to steps S3 and S4, a final accumulation matrix is ​​obtained, and logarithmic mapping and normalization processing are performed on the values ​​in the final accumulation matrix to generate a grayscale event frame.

[0011] Optionally, the fitting gradient and likelihood of the current event relative to each historical event in the spatiotemporal neighborhood of the historical matrix are obtained, and the two best historical events are selected to calculate the spatiotemporal gradient of the current event.

[0012] Optionally, the specific steps for selecting two optimal historical events to calculate the spatiotemporal gradient of the current event are as follows:

[0013] Get the relative gradient of current events and historical events;

[0014] Get the fitted gradient of the current event;

[0015] Based on the relative gradients of the current event and historical events, as well as the fitted gradient of the current event, the probability that each historical event and the current event are triggered by the same real-world point is obtained;

[0016] For all historical events in the spatiotemporal neighborhood, the two historical events with the highest probability are selected, and the average of their fitted gradients is taken as the spatiotemporal gradient of the current event.

[0017] Optionally, the relative gradient of the current event and the historical events The expression is:

[0018]

[0019] in, Indicates the current event c x Axis pixel coordinates, y Axis pixel coordinates and timestamps, Indicates the historical event n being fitted x Axis pixel coordinates, y Axis pixel coordinates and timestamps.

[0020] Optionally, the fitted gradient of the current event The expression is:

[0021] when hour: ;

[0022] when hour: ;

[0023] in, Represents the spatiotemporal gradient of historical event n.

[0024] Alternatively, the probability C of each historical event and the current event being triggered by the same real-world point is expressed as:

[0025]

[0026] in, e Represents a natural constant.

[0027] Optionally, the expression of the spatiotemporal gradient G of the current event is:

[0028]

[0029] in, and Represent the fitted spatiotemporal gradients of the two historical events s0 and s1 with the highest probability C respectively.

[0030] Optionally, the expression for the fitting starting point of the current event is:

[0031]

[0032] in, Respectively represent the fitting starting point of the current event c x Axis pixel coordinates, y axis pixel coordinates and timestamps; and Represents the starting points of the two historical events s0 and s1 with the highest probability C x Axis pixel coordinates, y Axis pixel coordinates and timestamps.

[0033] Optionally, the specific steps of step S4 are as follows:

[0034] Step S4-1: Obtaining the sub-pixel corrected coordinates of the current event based on the spatiotemporal coordinates of the current event and the spatiotemporal coordinates of the fitting starting point;

[0035] Step S4-2: Map the sub-pixel level corrected coordinates of each current event to the integer pixel coordinates of the accumulation matrix, and perform accumulation processing on the corresponding integer pixel coordinates to obtain an updated accumulation matrix with added weight values.

[0036] Optionally, the specific steps of step S5 are as follows:

[0037] Step S5-1: After all events in the segment are processed according to Step S3 and Step S4, a final cumulative matrix is ​​obtained, and logarithmic mapping is performed on the values ​​in the final cumulative matrix to obtain a normalized matrix;

[0038] Step S5-2: setting a preset threshold, and performing filtering on the normalized matrix to obtain a filter matrix;

[0039] Step S5-3: Generate an alignment event frame according to the filter matrix.

[0040] Compared with the prior art, the present invention has at least the following beneficial effects:

[0041] (1) The alignment method used in this invention significantly overcomes the reliance of traditional alignment methods on camera pose estimation, and is therefore more adaptable and flexible. In various scenarios, including those with a fixed camera, those with a moving camera, and those with a static or dynamic subject, this method can achieve efficient and high-precision alignment event frame generation, greatly expanding its scope of application.

[0042] (2) The alignment method adopted by the present invention can process event data more effectively. Through the spatiotemporal gradient propagation mechanism, motion blur and some noise in the event data can be effectively removed during the alignment process, thereby improving the generation quality of event frames.

[0043] (3) The alignment method employed in this invention provides a practical and feasible data preprocessing paradigm for various applications of event cameras, especially those combined with frame cameras. By optimizing the processing flow of event data, this invention can significantly improve the performance of various vision tasks and promote the widespread application of event cameras in vision systems. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] The drawings are only for purposes of illustrating particular embodiments and are not to be considered limiting of the invention.

[0045] Figure 1 This is a flow chart of the event frame image alignment method based on spatiotemporal gradient propagation of the present invention.

[0046] Figure 2 Schematic diagram of the original event frame when the spatiotemporal gradient propagation method is not used for alignment.

[0047] Figure 3 Schematic diagram of event frames after alignment according to the event frame image alignment method based on spatiotemporal gradient propagation of the present invention. DETAILED DESCRIPTION

[0048] In order to more clearly understand the above-mentioned objects, features and advantages of the present invention, the present invention is further described in detail below with reference to the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein. Therefore, the scope of protection of the present invention is not limited by the specific embodiments disclosed below.

[0049] One specific embodiment of the present application, as shown in Figure 1-Figure 3 , provides a spatiotemporal gradient propagation-based event frame image alignment method, and the specific steps are as follows:

[0050] Step S1, dividing the event stream data shot by the event camera into multiple event stream segments.

[0051] Specifically, the event stream data is data for recording dynamic visual information, and the form of the event stream data is an event sequence, which contains multiple events arranged in chronological order, and each event contains a parameter vector x , y , p , t ] wherein x and y are the x axis pixel coordinates and y axis pixel coordinates of the event, representing the position of the event in the image, p is the polarity of the event, representing the change in brightness, p a value of 1 indicates an increase in brightness (positive polarity), p a value of 0 indicates a decrease in brightness (negative polarity), t is the timestamp of the event, representing the time when the event occurs.

[0052] The present application adopts a segmented processing method to divide the events in the event stream data into multiple windows according to the timestamps.

[0053] Specifically, the event stream data shot by the event camera and the overall time range of the event stream data are obtained, including the start time and the end time;

[0054] determining a time window starting from the start time, the overall time range is evenly divided into multiple time windows based on the time window ;

[0055] For each event in the event stream data, the event is assigned to the corresponding time window according to the timestamp of the event, and multiple event stream segments are generated.

[0056] In some embodiments, the positive and negative polarities of the events are classified and processed, and two alignment event frames of different polarities are generated, or an alignment event frame that distinguishes positive and negative polarities through grayscale is generated.

[0057] Step S2, constructing a history matrix M h and a cumulative matrix M f with the same size as the image resolution.

[0058] Specifically, the history matrix Mh Each pixel coordinate corresponds to a linear list. Events can be stored in the cells of the list corresponding to the pixel coordinates, with each cell storing an event. The content of each event includes the timestamp t of the event, the spatiotemporal gradient G of the event, and the spatiotemporal coordinates (x, y, t) of the starting point of the event;

[0059] Furthermore, the spatiotemporal gradient of an event is: the time differential of the pixel displacement of the real-world point corresponding to the event in the event camera coordinate system. This gradient represents the displacement speed of the real-world point corresponding to the event on the image plane. The world point refers to the real physical world point that triggers the event. For example, when photographing a table, the corner of the table is a real-world point, which will trigger multiple events during the photographing process. In the records of the event camera, the timestamps and coordinates of these events are different, but they all correspond to the same world point (the corner of the table).

[0060] Furthermore, the spatiotemporal coordinates of the starting point of an event are: the pixel coordinates of the real-world point corresponding to the event in the event camera coordinate system at the starting moment of the event stream segment, and the timestamp of the corresponding event stream segment starting moment under the camera clock.

[0061] Specifically, the accumulation matrix M f Each pixel coordinate is used to record the accumulation of event points aligned to the pixel coordinate, and the accumulation matrix M f Finally, it is used to generate aligned event frame images.

[0062] Step S3: Process the events in the event stream segment one by one, according to the history matrix M h Historical events in the spatiotemporal neighborhood are used to obtain the spatiotemporal gradient of the current event; the fitting starting point is obtained based on the spatiotemporal gradient of the current event and saved in the historical matrix M h middle.

[0063] Furthermore, the current event is calculated relative to the historical matrix M h The fitting gradient G of each historical event in the spatiotemporal neighborhood fit and possibility C, and select the two best historical events to calculate the spatiotemporal gradient and starting point of the current event.

[0064] It is understood that the specific event being processed is called the current event; the processed event is the historical event and is stored in the history matrix M h middle.

[0065] Specifically, obtain the relative gradient of the current event and the historical event , the expression is:

[0066]

[0067] in, Represents the time and space coordinates of the current event c (corresponding to the current event c x Axis pixel coordinates, y axis pixel coordinates and timestamps), Indicates the time and space coordinates of the historical event n being fitted (corresponding to the time and space coordinates of the historical event n being fitted x Axis pixel coordinates, y axis pixel coordinates and timestamps).

[0068] The fitted gradient of the current event The expression is:

[0069] when hour: ;

[0070] when hour: ;

[0071] in, Represents the spatiotemporal gradient of historical event n.

[0072] In some embodiments, linear, cubic, multi-time, etc. motion models are used to fit the spatiotemporal gradient of the current event.

[0073] Furthermore, the probability C of each historical event and the current event being triggered by the same real-world point is obtained, which is expressed as:

[0074]

[0075] in, e Represents a natural constant.

[0076] In some embodiments, the likelihood is calculated using similarity methods such as Jaccard distance, NCC correlation, and cosine similarity.

[0077] Furthermore, for all historical events in the spatiotemporal neighborhood, the two historical events with the highest probability C are selected, and the average value of their fitted gradients is taken as the spatiotemporal gradient G of the current event, which is expressed as:

[0078]

[0079] in, and They represent the fitted spatiotemporal gradients of the two historical events with the highest probability C.

[0080] Furthermore, for all historical events in the spatiotemporal neighborhood, the two historical events with the highest probability C are selected, and the midpoint of their starting points is taken as the fitting starting point of the current event. The expression is:

[0081]

[0082] in, Indicates the time and space coordinates of the fitting starting point of the current event c (corresponding to the time and space coordinates of the current event c x Axis pixel coordinates, y axis pixel coordinates and timestamps); and The table shows the time-space coordinates of the starting points of the two historical events with the highest probability C (corresponding to the time-space coordinates of the two historical events with the highest probability C, s0 and s1). x Axis pixel coordinates, y axis pixel coordinates and timestamps).

[0083] In some embodiments, the spatial-temporal coordinates of the starting point of the current event are calculated by using the historical event with the highest probability C alone, the top N historical events with the highest probability C, performing arithmetic averaging, or performing weighted averaging according to the exponential power of the probability C.

[0084] Furthermore, the spatiotemporal neighborhood of the current event is: the history matrix M h Among all events stored in , the Manhattan distance to the pixel coordinates of the current event is less than the preset threshold r, and the timestamp difference is less than the preset threshold t max The events of constitute the spatiotemporal neighborhood of the current event.

[0085] Step S4: Calculate the corrected coordinates of the corresponding event based on the spatiotemporal coordinates of each event, the spatiotemporal coordinates of the fitting starting point, and the alignment time to obtain an updated cumulative matrix;

[0086] Step S4-1: According to the spatiotemporal coordinates (x s ,y s ,t s ) and space-time coordinates (x c ,y c ,t c ), obtain the sub-pixel corrected coordinates (x′, y′) of the current event c, expression:

[0087]

[0088] .

[0089] in, The start timestamp of the event stream segment.

[0090] Step S4-2: Correct the coordinates (x′, y′) of each current event c according to the sub-pixel level (i.e.: x Axis sub-pixel level correction coordinates, y Axis sub-pixel corrected coordinates), mapped to the accumulation matrix M fThe integer pixel coordinates of the image are accumulated by using the distance-based weight distribution method to obtain an updated accumulation matrix with added weight values.

[0091] Step S4-2-1, selecting multiple integer pixel coordinates closest to the sub-pixel corrected coordinate (x′, y′);

[0092] Preferably, the four integer pixel coordinates closest to the sub-pixel corrected coordinates are selected.

[0093] Step S4-2-2: Calculate the weight corresponding to each integer pixel coordinate using bilinear interpolation based on the difference between the sub-pixel corrected coordinate and the plurality of integer pixel coordinates;

[0094] Step S4-2-3: Accumulate the weights corresponding to the multiple integer pixel coordinates in step S4-2-2 into the accumulation matrix M f The corresponding pixel coordinate value is obtained by adding the weight value to update the accumulation matrix.

[0095] Step S5: After all events in the segment are processed according to steps S3 and S4, a final accumulation matrix is ​​obtained, and logarithmic mapping and normalization processing are performed on the values ​​in the final accumulation matrix to generate a grayscale event frame.

[0096] Step S5-1: After processing all events in the fragment according to steps S3 and S4, the final accumulation matrix is ​​obtained M f , for the final cumulative matrix M f The values ​​in are logarithmically mapped to obtain the normalized matrix , to enhance the contrast and reduce the impact of extreme values. Subsequently, the result is normalized so that its value range is mapped to [0,1] and converted into a grayscale value that conforms to the standard grayscale image pixel range. The expression is:

[0097]

[0098] ;

[0099] in, Represents the final cumulative matrix M f Logarithmic matrix after logarithmic processing; Represents the logarithmic result The normalized matrix after normalization.

[0100] Step S5-2: Set a preset threshold T and normalize the matrix Perform filtering to obtain the filter matrix , to suppress noise and highlight valid event information.

[0101] According to the normalized matrix The mid-point ( x , y ) coordinate values , calculate the filter matrix The coordinate value of the corresponding coordinate point , the expression is:

[0102] .

[0103] Step S5-3: According to the filter matrix Generate a frame of aligned events.

[0104] It can be understood that according to the filter matrix Generate an aligned event frame obtained through format conversion.

[0105] See also Figure 2 and Figure 3 , we can see that the aligned event frames become significantly clearer, the outlines of the captured objects in the image can be seen clearly, and some noise has been removed.

[0106] The above description is only a preferred specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any changes or substitutions that can be easily thought of by any technician familiar with this technical field within the technical scope disclosed by the present invention should be covered by the scope of protection of the present invention.

Claims

1. An event frame image alignment method based on spatiotemporal gradient propagation, characterized in that: The specific steps are as follows: Step S1: Divide the event stream data captured by the event camera into multiple event stream segments; Step S2: construct a history matrix and a cumulative matrix of the same size as the image resolution; Step S3: Process the events in the event stream segment one by one, obtain the spatiotemporal gradient of the current event based on the historical events in the spatiotemporal neighborhood of the history matrix; obtain the fitting starting point of the current event based on the spatiotemporal gradient of the current event and save it in the history matrix; Among them, the fitting gradient and probability of the current event relative to each historical event in the spatiotemporal neighborhood of the historical matrix are obtained, and the two best historical events are selected to calculate the spatiotemporal gradient of the current event; The specific steps for selecting two optimal historical events to calculate the spatiotemporal gradient of the current event are as follows: Get the relative gradient of current events and historical events; Get the fitted gradient of the current event; Based on the relative gradients of the current event and historical events, as well as the fitted gradient of the current event, the probability that each historical event and the current event are triggered by the same real-world point is obtained; For all historical events in the spatiotemporal neighborhood, the two historical events with the highest probability are selected, and the average value of their fitted gradients is taken as the spatiotemporal gradient of the current event; Among them, the relative gradient of current events and historical events The expression is: in, Indicates the current event c x Axis pixel coordinates, y Axis pixel coordinates and timestamps, Indicates the historical event n being fitted x Axis pixel coordinates, y axis pixel coordinates and timestamps; The fitted gradient of the current event The expression is: when hour: ; when hour: ; in, represents the spatiotemporal gradient of historical event n; the probability C that each historical event and the current event are triggered by the same real-world point is expressed as: in, e represents a natural constant; the expression of the space-time gradient G of the current event is: in, and Represent the fitted spatiotemporal gradients of the two historical events s0 and s1 with the highest probability C respectively; Step S4: Calculate the corrected coordinates of the corresponding event based on the spatiotemporal coordinates of each event, the spatiotemporal coordinates of the fitting starting point, and the alignment time to obtain an updated cumulative matrix; Step S5: After all events in the event stream segment are processed according to steps S3 and S4, a final accumulation matrix is ​​obtained, and logarithmic mapping and normalization processing are performed on the values ​​in the final accumulation matrix to generate a grayscale event frame.

2. The event frame image alignment method according to claim 1, characterized in that: The expression of the fitting starting point of the current event is: in, Respectively represent the fitting starting point of the current event c x Axis pixel coordinates, y axis pixel coordinates and timestamps; and Represents the starting points of the two historical events s0 and s1 with the highest probability C x Axis pixel coordinates, y Axis pixel coordinates and timestamps.

3. The event frame image alignment method according to claim 2, characterized in that: The specific steps of step S4 are as follows: Step S4-1: Obtaining the sub-pixel corrected coordinates of the current event based on the spatiotemporal coordinates of the current event and the spatiotemporal coordinates of the fitting starting point; Step S4-2: Map the sub-pixel level corrected coordinates of each current event to the integer pixel coordinates of the accumulation matrix, and perform accumulation processing on the corresponding integer pixel coordinates to obtain an updated accumulation matrix with added weight values.

4. The event frame image alignment method according to claim 3, characterized in that: The specific steps of step S5 are as follows: Step S5-1: After all events in the segment are processed according to Step S3 and Step S4, a final cumulative matrix is ​​obtained, and logarithmic mapping is performed on the values ​​in the final cumulative matrix to obtain a normalized matrix; Step S5-2: setting a preset threshold, and performing filtering on the normalized matrix to obtain a filter matrix; Step S5-3: Generate an alignment event frame according to the filter matrix.

Images