Dynamic data optimization and representation method based on asynchronous event stream

By optimizing the event camera's data representation through a Markov random field-based event energy model and dynamic time chamber span adjustment, the problem of poor imaging performance in multi-object scenes is solved, and clearer image processing is achieved.

CN118741336BActive Publication Date: 2024-11-26CHANGCHUN INST OF OPTICS FINE MECHANICS & PHYSICS CHINESE ACAD OF SCI
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202411221166.6
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2024-09-02
Publication Date
2024-11-26
Estimated Expiration
2044-09-02

AI Technical Summary

Technical Problem

Existing data processing methods based on event cameras cannot achieve perfect slicing when processing multiple objects with different moving speeds, and suffer from information redundancy and noise problems, resulting in poor imaging effects.

Method used

We adopt an event energy model based on Markov random fields to divide the event flow into main event flow and non-main event flow. By dynamically adjusting the time bin span and polarity attribute, we optimize the representation method of the event flow, remove redundant information and supplement missing information.

Benefits of technology

It effectively removes noise and redundant information from the event stream, improving the clarity and accuracy of the image, especially performing well in scenes with multiple objects at different speeds.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118741336B_ABST
    Figure CN118741336B_ABST
Patent Text Reader

Abstract

The present invention relates to the field of image processing technology, and in particular to a method for dynamic data optimization and representation based on asynchronous event streams. The method comprises: dividing the original event stream of an event camera into a main event stream and a non-main event stream according to the amount of main information carried; constructing an event energy model based on a Markov random field by establishing a complete energy function; redefining the main event stream as S i and the non-subject event stream is defined as S j ; According to the default time span, the main event stream S i and non-subject event stream S j Split into several time bins; adjust the length of the time bin span, then match the time bins one by one according to the sequence, and finally merge them into a new event stream; according to the main event stream S i The correlation with the original event stream L is used to repair the main information in the new event stream, and the redundant events in the time-space correlation area of ​​the new event stream are deleted to achieve information optimization. The advantages are: removing noise events in the event stream and ensuring the slicing effect.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a dynamic data optimization and representation method based on asynchronous event stream. Background Art

[0002] As a bionic visual sensor, event cameras have shown great advantages in terms of high dynamic range and high temporal resolution for visual tasks, providing a basis for efficient and reliable motion estimation in the most complex scenes. This sensor, also known as an event-based sensor, is a new type of bionic visual device inspired by the structure of the biological retina. However, these advantages come at a price, such as the low resolution of existing event-based visual sensors, and the redundancy and incompleteness of information data in the process of event representation. And due to the characteristics of the event stream data itself, we cannot use it directly, and we need to perform preprocessing steps of slicing and framing first. In general, there are now a variety of preprocessing algorithms that can achieve slicing and framing of event streams, but when there are multiple subjects with changing and different motion speeds in the event stream, the existing processing methods cannot achieve good results, and may also amplify the existing defects of the above-mentioned event information stream itself.

[0003] The output data of an event camera consists of a series of events, which encode the time, position and polarity information of the brightness change of a certain pixel. This is the advantage of event cameras, but it also means that each event carries very little information about the scene. Therefore, before interpreting or using the information on the event stream, we need to aggregate the information to proceed to the next step of information processing. In the field of event-based vision, the existing representation methods can be roughly divided into two types: sparse representation and dense representation. The method of using sparse representation can preserve the sparsity in the event, but the existing hardware equipment and subsequent data processing algorithms are not mature and targeted enough, and the event data preprocessed by this type of representation method cannot be extended to more complex tasks. Specifically, the sparse representation event data required by asynchronous spike neural networks is limited by the lack of specialized hardware and computationally efficient backpropagation algorithms. In addition, when processing the spatiotemporal characteristics of event data, point cloud encoders are indispensable, but the computational cost is high and there is a lot of noise information after processing. In addition, graph neural networks have a fairly high scalability and can achieve fairly high performance on most visual tasks. However, in terms of event-based vision, the accuracy of these methods is still not as good as dense methods.

[0004] Relatively speaking, dense representations have better performance because they can match existing mature machine learning algorithms and neural network architectures. Most of the early dense representation methods converted events into histograms, time surfaces, or a combination of the two, and then used standard image processing algorithm models for subsequent processing. However, since they usually only use a few channels, the results obtained by these methods only capture low-dimensional representations of events. In further studies, researchers tried to capture more event information by calculating high-order moments or stacking multiple time windows of events.

[0005] At present, there are some related solutions for event data stream representation methods at home and abroad. The prior art proposes an asynchronous event-to-frame conversion algorithm, which asynchronously distorts the spatiotemporal information in retinal events into a new adaptive time plane linear time decay (ATSLTD) frame representation, such as Figure 1 As shown. The conversion is driven by object motion: (1) Fast object motion generates more ATSLTD frames than slow object motion, creating a relatively large spatiotemporal space for object tracking; (2) If the target object does not have a large enough motion (displacement), no ATSLTD frames are generated, thereby increasing the calculation speed. Since intensity changes and retinal events usually occur at the edges of the target object, the ATSLTD frames record clear and sharp object contours. However, existing methods are essentially still based on a fixed time window to complete event stacking. When the event rate becomes too large or too small, these methods will cause some problems. And due to the sparsity and asynchronous nature of the event camera itself, the existing classical computer vision algorithms can achieve good results, but they are not completely applicable. Summary of the invention

[0006] In order to solve the above problems, the present invention provides a dynamic data optimization and representation method based on asynchronous event stream.

[0007] The present invention aims to provide a method for optimizing and representing dynamic data based on asynchronous event streams, which specifically comprises the following steps:

[0008] S1. Divide the original event stream L of the event camera into the main event stream U and the non-main event stream N according to the amount of main information carried;

[0009] S2. By establishing the complete energy function corresponding to the main event flow U and the original event flow L and the complete energy function corresponding to the non-main event flow N and the original event flow L, an event energy model based on Markov random field is constructed; after constructing the model, the main event flow is redefined as S i , define the non-subject event stream as S j ;

[0010] S3. The main event stream S is converted into a time span according to the default time span. i and non-subject event stream S j Split into several time bins; adjust the length of the time bin span, match the time bins one by one according to the sequence, and finally merge them into a new event stream;

[0011] S4. According to the main event flow S i The correlation with the original event stream L is used to repair the part of the new event stream where the main information is located, and the redundant events in the time-space correlation area of ​​the new event stream are deleted to achieve information optimization.

[0012] Preferably, in step S3, a dynamic asynchronous data measurement and slicing algorithm is used to adjust the length of the time bin span.

[0013] Preferably, the dynamic asynchronous data measurement and slicing algorithm specifically includes: taking the polarity attribute of the event in the corresponding time bin in the event stream as the basis for calculating the overall density, and introducing the new event density into the time window as the basic reference basis, and dynamically adjusting the length of the time bin span; the adjustment of the time bin span is achieved by updating the time span, and the formula is as follows:

[0014]

[0015]

[0016] In the formula, represents the time span of each time bin, and Represent the main event stream S i and non-subject event stream S j The time bin in and Represents two time bins and The time span before the update, and Represents two time bins respectively and The updated time span of and Represent the three-dimensional Gaussian kernel belonging to the i-th time bin and the j-th time bin respectively. , and represents the sum of the squared moduli of all time bins of the event stream, represents a space-time interval, Represents two time bins and The distance between them is given by the following formula:

[0017]

[0018] in, express and The inner product of represents the inner product of the ith time bin, represents the inner product of the j-th time bin.

[0019] Preferably, step S1 uses a main area recognition algorithm for division; when performing area judgment and division operations on the event stream, an improved Pearson correlation coefficient is used. and Spearman correlation coefficient , the formulas are as follows:

[0020] ;

[0021] ;

[0022] in, Represents the horizontal axis coordinate of the i-th event; Represents the vertical axis coordinate of the i-th event; and It represents the weighted average of other events on the horizontal and vertical axes at the time node where the i-th event occurs, with the frequency of the event as the weight.

[0023] Preferably, the complete energy function corresponding to the main event stream U and the original event stream L in step S2 is The expression is as follows:

[0024] ;

[0025] in, The energy function expressions are:

[0026] ;

[0027] ;

[0028] ;

[0029] ;

[0030] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the main event stream U and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the main event stream U; α and β are non-negative dynamic weight parameters corresponding to the largest clusters.

[0031] Preferably, the complete energy function corresponding to the non-main event stream N and the original event stream L in step S2 is The expression is as follows:

[0032] ;

[0033] in, The energy function expressions are:

[0034] ;

[0035] ;

[0036] ;

[0037] ;

[0038] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the non-subject event stream N and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the non-main event stream N; α and β are non-negative dynamic weight parameters corresponding to the largest clusters.

[0039] Preferably, α reflects the spatial correlation of events, and the expression is as follows:

[0040] ;

[0041] The β reflects the temporal correlation of events, and the expression is as follows:

[0042] ;

[0043] In the above formula, Respectively represent the pixel difference between two events in the energy function in the horizontal and vertical directions, Respectively represent the maximum difference of events in all the largest clusters in the horizontal and vertical directions, and Respectively represent in all the largest groups The minimum and maximum time intervals between events.

[0044] Preferably, the method for patching the part of the new event stream where the main information is located in step S4 is as follows: the local memory time plane of the new event stream is used as the basic operator, and adjacent pixels are grouped into units of the same size and number; using the formula Calculate the initial warp field of each unit and update the warp field range in a gradient descent manner to obtain the optimal unit compensation event density and regional scope The value of is used to determine the event range in the original event stream L that complements the main body in the new event stream and to repair it;

[0045] In the formula, x represents the horizontal coordinate of the pixel corresponding to the event, y represents the vertical coordinate of the pixel corresponding to the event, E x and E y represents the local error in the plane displacement during the main event compensation process, E t Indicates the local error in the time axis during the main event compensation process. It represents the angle between the compensation event cloud after modeling with x, y, and t as the three axes and the z-axis represented by t, and t represents the time corresponding to the event.

[0046] Compared with the prior art, the present invention can achieve the following beneficial effects:

[0047] (1) In the process of dividing the original event stream into the main event stream and the non-main event stream, the events are classified according to the overall correlation between the events and the event streams. This can identify some noise events that are difficult to remove with existing methods and have strong correlations between events themselves, such as diffuse reflection noise.

[0048] (2) The probability problem is transformed into an energy problem. Based on the fact that the joint probability distribution can be expressed as the product of functions on their largest cluster, an energy function is defined that can describe the relationship between different events after quantization. The smaller the value of the energy function, the smaller the energy and the higher the probability, indicating that the correlation between the two events is greater. This definition of event correlation is closer to the relationship between the events themselves than the existing definition method.

[0049] (3) The polarity of the events in the corresponding peak data bin in each event stream is used as the basis for calculating the overall density, and the new event density is introduced into the time window as the basic reference basis to dynamically adjust the length of the time bin span; this method is more adaptable to complex shooting situations than the existing method based on fixed time or fixed number of events.

[0050] (4) Taking the main event stream as the basis, the impact of the redundant non-main event stream with low correlation around the corresponding main space in the new event stream is minimized; then, for the missing main events in the time bin, events with high correlation are selected to supplement them; this method can well handle the problem that the existing method cannot achieve perfect slicing effect when there are two or more objects with significant speed differences in the scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a schematic diagram of an event camera.

[0052] Figure 2 It is a schematic diagram of spatial correlation and temporal correlation between events provided according to an embodiment of the present invention.

[0053] Figure 3 is the maximum cluster corresponding to each energy function provided according to an embodiment of the present invention.

[0054] Figure 4 It is a single neuron spike train measurement result provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0055] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, the same modules are represented by the same reference numerals. In the case of the same reference numerals, their names and functions are also the same. Therefore, the detailed description thereof will not be repeated.

[0056] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and do not constitute a limitation of the present invention.

[0057] Summary of the invention: The purpose of the present invention is to solve the technical problem that the existing method cannot achieve a perfect slicing effect when there are two or more objects with significant speed differences in the scene, and on this basis, to remove as much redundant information as possible, retain more pre-processed data related to real events and subjects, so that the final imaging effect is clearer and more restored.

[0058] In order to make full use of the spatiotemporal structural characteristics and polarity properties of the event stream itself, we proposed a dynamic asynchronous data optimization and representation algorithm. The algorithm first divides the event stream into a main event stream and a non-main event stream, and then uses Markov random fields to quantify the correlation and importance between each event and between the event and the event stream to determine which are the main events that need to be retained and which are the non-main events that carry unimportant information; then the two event streams are divided into time bins according to the default time span; then the polarity of the events in the corresponding time bins in each event stream is used as the basis for calculating the overall density, and the new event density is introduced into the time window as the basic reference basis to dynamically adjust the length of the time bin span; after completing the adjustment of the length of the time bins in the two event streams, the time bins are matched one by one according to the sequence, and finally merged into a new event stream. After adjusting the span corresponding to the time bin to output a new event stream, the algorithm will cause loss and redundancy of events within each time bin. Based on this situation, we defined a new adaptive spatiotemporal main surface transformation algorithm. The algorithm can use the main event stream as the basis to minimize the impact of the redundant non-main event streams with low correlation around the corresponding main space in the new event stream. Then, for the missing main events in the time bin, the highly correlated events are selected to supplement them.

[0059] The present invention provides a method for optimizing and representing dynamic data based on asynchronous event streams, which specifically comprises the following steps:

[0060] S1. Divide the original event stream L of the event camera into the main event stream U and the non-main event stream N according to the amount of main information carried;

[0061] Specifically, the main area recognition algorithm is used for division; when performing area judgment and division operations on the event stream, the improved Pearson correlation coefficient is used and Spearman correlation coefficient , the formulas are as follows:

[0062] ;

[0063] ;

[0064] in, Represents the horizontal axis coordinate of the i-th event; Represents the vertical axis coordinate of the i-th event; and It represents the weighted average of other events on the horizontal and vertical axes at the time node where the i-th event occurs, with the frequency of the event as the weight.

[0065] S2. By establishing the complete energy function corresponding to the main event flow U and the original event flow L and the complete energy function corresponding to the non-main event flow N and the original event flow L, an event energy model based on Markov random field is constructed; after constructing the model, the main event flow is redefined as S i , define the non-subject event stream as S j ;

[0066] The complete energy function corresponding to the main event flow U and the original event flow L The expression is as follows:

[0067] ;

[0068] in, The energy function expressions are:

[0069] ;

[0070] ;

[0071] ;

[0072] ;

[0073] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the main event stream U and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the main event stream U; α and β are non-negative dynamic weight parameters corresponding to the largest cluster;

[0074] α reflects the spatial correlation of events, and its expression is as follows:

[0075] ;

[0076] β reflects the temporal correlation of events, and its expression is as follows:

[0077] ;

[0078] In the above formula, Respectively represent the pixel difference between two events in the energy function in the horizontal and vertical directions, Respectively represent the maximum difference of events in all the largest clusters in the horizontal and vertical directions, and Respectively represent in all the largest groups The minimum and maximum time intervals between events.

[0079] The complete energy function corresponding to the non-main event stream N and the original event stream L The expression is as follows:

[0080] ;

[0081] in, The energy function expressions are:

[0082] ;

[0083] ;

[0084] ;

[0085] ;

[0086] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the non-subject event stream N and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the non-main event stream N; α and β are non-negative dynamic weight parameters corresponding to the largest clusters.

[0087] S3. The main event stream S is converted into a time span according to the default time span. i and non-subject event stream S j Divide into several time bins; use dynamic asynchronous data measurement and slicing algorithm to adjust the length of the time bin span, and then match the time bins one by one according to the sequence after adjustment, and finally merge them into a new event stream;

[0088] Specifically, the Asynchronous Spike DynamicMetric and Slicing algorithm (ASDMS) is used to take the polarity attributes of events in the corresponding time bins in the event stream as the basis for calculating the overall density, and the new event density is introduced into the time window as the basic reference basis to dynamically adjust the length of the time bin span; after completing the adjustment of the length of the time bin in the event stream, the time bins are matched one by one according to the sequence, and finally merged into a new event stream;

[0089] Adjusting the time bin span is achieved by updating the time span, the formula is as follows:

[0090]

[0091]

[0092] In the formula, represents the time span of each time bin (in microseconds), and Represent the main event stream S i and non-subject event stream S j The time bin in and Represents two time bins and The time span before the update, and Represents two time bins respectively and The updated time span of and Represent the three-dimensional Gaussian kernel belonging to the i-th time bin and the j-th time bin respectively. , and represents the sum of the squared moduli of all time bins of the event stream, represents a space-time interval, Represents two time bins and The distance between them is given by the following formula:

[0093]

[0094] in, express and The inner product of represents the inner product of the i-th time bin (that is, the square of the modulus of the time span of the time bin), represents the inner product of the jth time bin (that is, the square of the modulus of the time span of that time bin).

[0095] S4. According to the main event flow S i The correlation with the original event stream L is used to repair the part of the new event stream where the main information is located, and the redundant events in the time-space correlation area of ​​the new event stream are deleted to achieve information optimization;

[0096] The method for patching the part of the new event stream where the main information is located is as follows: the local memory time surface of the new event stream is used as the basic operator, and adjacent pixels are grouped into units of the same size and number; using the formula Calculate the initial warp field of each unit and update the warp field range in a gradient descent manner to obtain the optimal unit compensation event density and regional scope The value of is used to determine the event range in the original event stream L that complements the main body in the new event stream and to repair it;

[0097] In the formula, x represents the horizontal coordinate of the pixel corresponding to the event, y represents the vertical coordinate of the pixel corresponding to the event, E x and E y represents the local error in the plane displacement during the main event compensation process, E t Indicates the local error in the time axis during the main event compensation process. It represents the angle between the compensation event cloud after modeling with x, y, and t as the three axes and the z-axis represented by t, and t represents the time corresponding to the event.

[0098] Example 1

[0099] This embodiment provides a method for optimizing and representing dynamic data based on an asynchronous event stream, which specifically includes the following steps:

[0100] S1. Using the subject area recognition algorithm, the event camera event stream is divided into the subject event stream and the non-subject event stream according to the amount of main information carried; the subject event stream Based on real events, non-main event flow Mainly noise events;

[0101] The present invention first uses the main area recognition algorithm to divide the original event stream L into the main event stream U mainly composed of real events and the non-main event stream N mainly composed of noise events. On the one hand, it identifies the unconventional noise events with strong correlation between its own events, and on the other hand, it is convenient for the algorithm used in the subsequent process to take the corresponding dynamic judgment threshold according to the different characteristics of the two event streams for further denoising operations.

[0102] When performing regional judgment and division operations on event streams, we proposed an improved Pearson correlation coefficient based on the relevant knowledge of Pearson correlation and Spearman correlation and the characteristics of the event stream itself. and Spearman correlation coefficient , the formula is as follows:

[0103]

[0104]

[0105] in Represents the horizontal axis coordinate of the i-th event; Represents the vertical axis coordinate of the i-th event; and It represents the weighted average of other events on the horizontal and vertical axes at the time node where the i-th event occurs, with the frequency of the event as the weight.

[0106] The Pearson correlation coefficient, also known as the product-moment correlation coefficient, is used to measure the linear correlation and normal distribution correlation between two random variables. Its value range is between [-1,1], -1 indicates a completely negative correlation, 1 indicates a completely positive correlation, and 0 indicates no correlation at all, which means that when the coefficient is 0, the changes in the two variables have no effect on each other. The Spearman correlation coefficient belongs to the rank correlation coefficient, and can reflect the correlation between the changing trend directions of the two random variables like the Pearson correlation coefficient, but the Spearman correlation coefficient measures the correlation between nonlinear relationship variables. When the two correlation coefficients correspond to When the value of is between 0.8 and 1, the correlation between the corresponding events is strongly correlated. The two improved correlation coefficient formulas can fully measure the correlation between events in the event stream, thereby dividing the original event stream L into the main event stream U dominated by real events and the non-main event stream N dominated by noise events.

[0107] S2. By establishing the complete energy function corresponding to the main event flow U and the original event flow L and the complete energy function corresponding to the non-main event flow N and the original event flow L, an event energy model based on Markov random field is constructed; after constructing the model, the main event flow is redefined as S i and the non-subject event stream is defined as S j ;

[0108] The complete energy function corresponding to the main event flow U and the original event flow L The expression is as follows:

[0109] ;

[0110] in, The energy function expressions are:

[0111] ;

[0112] ;

[0113] ;

[0114] ;

[0115] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the main event stream U and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the main event stream U; α and β are non-negative dynamic weight parameters corresponding to the largest cluster;

[0116] α reflects the spatial correlation of events, and its expression is as follows:

[0117] ;

[0118] β reflects the temporal correlation of events, and its expression is as follows:

[0119] ;

[0120] In the above formula, Respectively represent the pixel difference between two events in the energy function in the horizontal and vertical directions, Respectively represent the maximum difference of events in all the largest clusters in the horizontal and vertical directions, and Respectively represent in all the largest groups The minimum and maximum time intervals between events.

[0121] Similarly, the complete energy function corresponding to the non-main event flow N and the original event flow L is obtained The expression is as follows:

[0122] ;

[0123] in, The energy function expressions are:

[0124] ;

[0125] ;

[0126] ;

[0127] ;

[0128] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the non-subject event stream N and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the non-main event stream N; α and β are non-negative dynamic weight parameters corresponding to the largest clusters.

[0129] Principle: The core idea of ​​Markov random field is Markov property, which means that the value of the current observed variable is affected by the current state and other states; that is, after a random variable state sequence is expanded in chronological order, given the current state and all past states, the conditional probability distribution of its future state depends only on the current state. In other words, when the current state of the random variable is given, its value is independent of the past state, then this random process has Markov property. The meaning of random field is that when a value in the phase space is randomly assigned to each position according to a certain distribution, the whole is called a random field. The combination of the two is Markov random field.

[0130] A model based on the concept of Markov random fields is a model consisting of nodes and connecting segments, which is used to describe the undirected interaction between variables (such as Figure 2 As shown in Figure 1, nodes represent variables, and connecting lines represent the probability of interaction between neighboring variables. Based on the above prior knowledge, the nodes in the model are used to represent events in the event stream, and connecting lines are used to represent the correlation between events by using the nature of the Markov random field model and the event stream of the event camera.

[0131] In step S1, the event stream is initially divided into a main event stream based on the relevance of the event to the overall event stream. , and non-subject event flows dominated by noise events ; Among these two events, the main event stream U is generated by the movement of the object, and the other non-main event stream N is generated by its own hardware and external interference. The combination of these two event streams is the original event stream To further filter out the noise events in the main event stream and the real events in the non-main event stream, it is necessary to combine the characteristics of the two event streams and design a distinction method that can be dynamically adjusted according to the situation and better reflect the correlation between events.

[0132] In most cases, the event stream is dominated by real events, which means that U and L have a strong correlation, and the correlation between N and L is relatively smaller. Based on this situation, we can combine the relevant properties of the Markov random field model to express the relationship between U and L and between N and L with a joint probability distribution, and then proceed to the next step based on this. According to the Hammersley-Clifford theorem, the joint probability distribution of the Markov random field model can be expressed as the product of non-negative functions of random variables on their largest clusters. This operation is also called the factorization of the Markov random field model as follows:

[0133] ;

[0134] In the formula, is the maximum clique in the model; in the Markov random field model, a subset of nodes in which any two nodes are connected by an edge is called a clique. If C is a clique in the model and no more nodes can be added to make it a larger clique, then C is called the maximal clique.

[0135] for The potential function on is defined in units of the largest cluster, that is, there are as many potential functions as there are largest clusters in a graph; the potential function is used to quantify the joint probability of random variables in the largest cluster, but it is not normalized. Since the potential function is restricted to be strictly greater than zero to ensure 0, so it is more convenient to express the potential function in exponential form, as follows:

[0136] ;

[0137] Z is the partition function, also called the normalization factor, which corresponds to the sum of the factorization results of all the largest groups in the Markov random field model. The product form is normalized to the probability form; the expression of Z is as follows:

[0138] ;

[0139] is the energy function within the corresponding group. We combine the above two equations and express the potential function in the joint probability distribution with the energy function. At this time, the joint probability distribution satisfies the Gibbs distribution:

[0140] ;

[0141] It is the complete energy function corresponding to the Markov random field model of event stream U and event stream L:

[0142]

[0143] In the Markov random field model we use, represents a single event in U, represents a single event in N, Represents a single event in L. Because there are only three states for pixels in the event stream: an event with brightness enhancement occurs ( =1); a brightness reduction event occurs ( =0); no event is generated. Based on this characteristic, we can define , , ,in They respectively represent the horizontal position information, vertical position information and timestamp information of the event.

[0144] And when For real events, There is a strong correlation between and other events in the spatial and temporal neighborhoods and There is also a strong correlation. When it is a noise event in U, the situation is completely opposite. and , and The correlation is not strong. They respectively represent the differences between other events and the original event in the neighborhood of the event in terms of horizontal position, vertical position and timestamp.

[0145] According to the above prior knowledge, the Markov random field model of event stream U and event stream L contains the following four maximum groups: , , , For the maximum group { }, in order to describe the relationship between event variables within a group, we define the corresponding energy function.

[0146]

[0147] In this energy function, when and When in the same state, the energy given by the energy function is lower, and the corresponding correlation is higher. and When in the opposite state, the energy given by the energy function is higher and the corresponding correlation is lower.

[0148] Similarly, for the other three largest groups { }, { }, { }, we also define the corresponding energy function as follows.

[0149]

[0150]

[0151]

[0152] α and β are non-negative dynamic weight parameters corresponding to the largest cluster. α reflects the spatial correlation of events as follows.

[0153]

[0154] β reflects the temporal correlation of events as follows.

[0155]

[0156] Represents the pixel difference between two events in the energy function in the horizontal and vertical directions respectively. They represent the maximum differences of events in the horizontal and vertical directions among all the largest clusters, respectively. Indicates that among all the largest groups, Minimum time interval with other events, Indicates the maximum time interval.

[0157] In summary, we can get the complete energy function corresponding to the main event flow U and the original event flow L :

[0158] ;

[0159] In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the main event stream U and the events in the original event stream L. The three largest groups represent the relationship between events in the spatiotemporal neighborhood of the main event stream U; the schematic diagram of the largest group is as follows Figure 3 shown.

[0160] Similarly, the complete energy function corresponding to the non-main event flow N and the original event flow L can be obtained: :

[0161] .

[0162] Combining the relationship between the above energy functions, we can get an event energy model based on Markov random fields. However, the event situation in the non-main event stream N with noise as the main body is different from that in the main event stream U. In step S1, when we distinguish the main event stream U and the non-main event stream N according to the overall correlation, random noise events such as BA noise with low correlation and unconventional noise events such as diffuse noise with high correlation are divided into the non-main event stream N. For the diffuse noise in the non-main event stream, although there is a strong correlation between the events in this noise area, they are not strongly correlated with the events in the original event stream L. In contrast, the events belonging to random noise in the non-main event stream N are not strongly correlated with the events in N or the events in the L event stream. The real events in the non-main event stream N have a strong correlation with the events in N and L. With these features as the basis for judgment, we can make a good distinction and remove the conventional noise events, unconventional noise events and real events in the event stream N. After completing these operations, we redefine the main event stream as S i , the non-subject event flow is defined as S j In this way, in the subsequent slicing process, we can obtain dynamic parameters representing the length of the slices that better reflect the relationship between our own data. In the following algorithm process, the event stream will be optimized and divided from two perspectives: spatiotemporal structure and polarity attributes.

[0163] S3. The main event stream S is converted into a time span according to the default time span. i and non-subject event stream S j The event stream is divided into several time bins. The Asynchronous Spike Dynamic Metric and Slicing algorithm (ASDMS) is used to calculate the overall density based on the polarity of events in the corresponding time bin in the event stream. The new event density is introduced into the time window as the basic reference to dynamically adjust the length of the time bin span. After the adjustment of the length of the time bin in the event stream is completed, the time bins are matched one by one in sequence and finally merged into a new event stream.

[0164] Principle description: In this step, an asynchronous spike data stream representing event information is given Equipped with a new spike distance metric , and proposed a new complete and separable metric space .when is a compact subset with Euclidean distance, It is also called a local finite configuration. , which means from a certain probability space to a measurable space. If ,but Represents a space-time marker point in a finite space. Probability space is a mathematical model of a random experiment, where the sample space is the set of all possible outcomes; and hour, represents a probability measure.

[0165] The process of quantifying asynchronous spikes is quite difficult due to the lack of standard algebraic operations. The kernel method can provide a general framework for the measurement process of spike sequences. This method can extend linear modeling in the input space to nonlinear modeling, especially mapping abstract objects to Hilbert space. This feature provides new solutions to key problems in the quantization process. Therefore, we choose to use the Gaussian kernel correlation method to measure the distance between asynchronous spikes in spatiotemporal markers. We let and Represents time and space intervals The inner product is introduced to measure the distance between asynchronous spikes in Hilbert space, and the formula is as follows:

[0166]

[0167] in, express and The inner product of represents the inner product of the ith time bin, represents the inner product of the j-th time bin.

[0168] When the process comes to analyzing the polarity properties of the statistical event stream, we find that the existing spike train measurement methods hardly use the polarity properties. Based on this, three representative pixels in the event camera are selected as the ordinate origin of the spike stream; the distance between spike trains is calculated by using the single neuron spike train measurement, and gradually increases with the increase of the pixel index. The measurement results are shown in Figure 4 As shown, it can be seen that the ON polarity ratio varies randomly. Therefore, the polarity attribute is used as the prior probability distribution in the spike metric, and the conditional intensity function is used to describe the spatiotemporal marker points, so that the polarity attribute is taken as the key point of calculation.

[0169] The conditional strength function expression is as follows:

[0170]

[0171]

[0172] In the formula, is the cumulative distribution function, It is the peak history The conditional probability density function in . Using the Bayesian principle to analyze the above formula, we can get the new expression of the conditional intensity function as follows:

[0173] ;

[0174] ; Choose to use a smoothing function to capture the spatiotemporal structure and use 3D convolution to transform discrete spikes into a continuous intensity function. It is similar to the historical time The conditional strength function in , but without considering polarity, is calculated as follows:

[0175]

[0176] Then, according to the historical time The polarity probability distribution in , and express it in the following form:

[0177] ;

[0178] Where Nu{} represents the space-time interval In addition, for any two spike data bins and , inner product It can be given by the following formula:

[0179] ;

[0180] Substituting into the previous formula, the formula can be transformed into the following form:

[0181] ;

[0182] ;

[0183] in and Respectively represent the corresponding peak data bins and The number of events in the spike bin and Indicates the event and , their inner product can be expressed as follows:

[0184] ;

[0185] When calculating the inner product, the Gaussian kernel function is used as the smoothing function and is defined as follows:

[0186]

[0187] In the formula, , is the standard deviation parameter of the Gaussian kernel, and , ; So it can be converted into the following form:

[0188]

[0189] Finally, the inner product between the two peak data bins can be further rewritten as:

[0190]

[0191]

[0192] in represents the relative polar static force in the spike data bin. In general, the optimal kernel parameter of the three-dimensional Gaussian kernel function involved in the above formula is We can continuously optimize by minimizing the fitting error. In order to obtain more accurate calculation results, we refer to the parameter structure of the compression ratio and calculate the performance score. The correlation coefficient between the peak metric distance d. Therefore, the error function corresponding to the optimal kernel function The definition is as follows:

[0193]

[0194] in is the peak metric distance between two peak event streams, and the corresponding sets are R and D respectively. yes A polynomial function of degree can fit the distance With performance scores In addition, Is a hyperparameter that weights the relative contribution of the norm penalty term to avoid overfitting. The minimum error is found by iterating the kernel parameters using the gradient method:

[0195]

[0196]

[0197] The learning rate is a hyperparameter that determines how we adjust the gradient of the loss. We set the learning rate and the maximum number of iterations The initial value of represents a positive number that approaches infinitely. Then we fit a polynomial function between the distance and the performance score , combined with the previous formula to update the kernel parameters , where n represents the iteration step with an initial value of 0. or When , the updating process stops, and the minimum error is obtained; otherwise, the value of n is increased once and the iteration continues. Then, the distance relationship between the spike data bin and its corresponding event stream needs to be calculated.

[0198] Substitute the calculated results into the following adaptive cubicle generation time calculation formula:

[0199]

[0200]

[0201] In the formula, represents the time span of each time bin (in microseconds), and Represent the main event stream S i and non-subject event stream S j The time bin in and Represents two time bins and The time span before the update, and Represents two time bins respectively and The updated time span of and Represent the three-dimensional Gaussian kernel belonging to the i-th time bin and the j-th time bin respectively. , and represents the sum of the squared moduli of all time bins of the event stream, represents a space-time interval, Represents two time bins and In this way, we can divide a continuous event stream into intervals of different time lengths according to time labels, and these intervals are allowed to be discontinuous or overlap with each other.

[0202] S4. According to the main event flow S i The correlation with the original event stream L is used to repair the part of the new event stream where the main information is located, and the redundant events in the time-space correlation area of ​​the new event stream are deleted to achieve information optimization.

[0203] In order to describe the impact of different events on each other at different times, the concept of time surface is introduced to describe the local spacetime around related events. The time surface can be transformed into a local space operator acting on the i-th event in the main event stream in a specific form, which is defined as follows:

[0204]

[0205] { 1,1}

[0206] in represents the radius of the spatial neighborhood used to compute the time surface, From the pixel is the time of the last event received with polarity q, and τ is a decay factor that gives less weight to events further in the past. Using a timetable encodes information about asynchrony in the event neighborhood, thus providing relationships between events in time and space.

[0207] In order to construct the feature event representation we need, we need to generalize the time surface first. The time surface only uses the time of the last event received near the pixel of the time surface. , which makes the descriptor too sensitive to non-major events or small changes in the original event stream. To address this shortcoming, use a time window with a span size adjusted in step S3 The time surface is calculated by the historical events in; a local memory time surface is defined as follows:

[0208]

[0209]

[0210] The above formula more effectively reduces the impact of non-subjective events and small changes in the event flow, while more robustly describing the real dynamics of the shooting scene. Then, referring to the idea of ​​obtaining invariance in terms of speed and contrast, the local memory time plane is used as the basic operator, and adjacent pixels are grouped into units of the same size and number. ; For each unit, we sum the time surface components calculated from the events in it into a histogram in the following form:

[0211]

[0212] in, The pixel coordinates of the current event are within the cell range. Since the event camera sensor is capturing, objects with higher contrast will generate more events than objects with lower contrast. For this feature, we normalize by counting the number of events within the spatiotemporal window. , so that the descriptor corresponding to the unit is more contrasting, and the histogram definition of the event group unit is obtained as follows:

[0213]

[0214] By regularizing space and time, the impact of non-main events on main events in the new event stream is weakened. Next, we need to make up for the main events that carry important information that were lost in the previous algorithm due to changes in time span.

[0215] Set time period Internal input in unit The events in the We use a warp field To express the time Mapping events at to their locations in time The next step is to find a corresponding motion compensation warp field. This allows the events we use to complement the event stream to have the most appropriate density when projected onto the image plane. We define these subject compensation events as:

[0216]

[0217] In the formula, represents the time projection function that projects the subject compensation event along the time axis, and Can be used to reduce the dimensionality of data: The warp field encodes information about each event in terms of the geometric properties of the event stream. So in the following we will represent it using two discrete maps that encode the intensity and time properties of the event stream. The data available in and To express.

[0218] To calculate the principal compensation event The event density , we discretize the image plane into pixel regions of a specified size. We use the notation To represent integer pixel coordinates, use Indicates the event coordinates after displacement. Each projection event of will be mapped to a discrete pixel, and the total number of events mapped to the pixel will be recorded as the value of the pixel, the formula is as follows:

[0219] ;

[0220] The above formula represents the event trajectory, which means a set of events along the time axis. Projected to pixels after operation And we define the event count image as:

[0221]

[0222] Cell compensation event density The calculation formula is as follows:

[0223] ;

[0224] in Indicates that at least one event is mapped to the unit The number of pixels within . During the projection operation, events generated by different subject edges may be projected onto the same pixel, which is contrary to the original intention of reducing interference events. To alleviate this problem, the time representation image obtained based on the timestamp information in the event is used to participate in the subject compensation process. The expression is as follows:

[0225] ;

[0226] In the formula, represents an event count image, is a discrete plane, each pixel contains a to its event timestamp. By averaging the timestamps, we can improve the fidelity of the main event stream by utilizing all available events. Existing methods only consider the latest timestamp, but since the signal-to-noise ratio of an event depends on the average illumination, the performance of this method is not ideal in low-light conditions. Following the 3D structure of the event cloud, its gradient can provide a global measure of the error in the main compensation process, which can be minimized as follows,

[0227]

[0228] Here and Respectively The local spatial gradient calculated using the Sobel operator along the X and Y axes. Given that this formula considers the global error of the event cloud within the warp field, we can combine it with the rigid body motion assumption and split it into the following equations:

[0229]

[0230]

[0231]

[0232]

[0233] The above four equations correspond to the local errors in the image plane compensation process in terms of displacement, expansion and two-dimensional rotation. We use a global subject event model that includes an error parameter To describe the corresponding global warp field , the resulting transformation is as follows:

[0234] ;

[0235] In the formula, x represents the horizontal coordinate of the pixel corresponding to the event, y represents the vertical coordinate of the pixel corresponding to the event, E x and E y represents the local error in the plane displacement during the main event compensation process, E t Indicates the local error in the time axis during the main event compensation process. It represents the angle between the compensation event cloud modeled with x, y, and t as the three axes and the z axis represented by t, where t represents the time corresponding to the event;

[0236] Through this transformation we transform the coordinates in the event stream Converted to , thus obtaining the expression of the warp field, and the timestamp will not be converted in this process. Based on the four-parameter model proposed above, we use the error function defined on the time representation image to estimate and minimize the motion range of the subject event flow, and then refine the global activity area of ​​the subject event by maximizing the subject event density on the event count image, find and track the event subset with high probability related to the subject in the non-subject event, and realize the complementation and motion compensation of the subject event flow.

[0237] The entire main event completion process is to first calculate the initial warp field expression of each unit C, and then generate its corresponding time representation image Using the initial model To calculate the model and its corresponding gradient, and update it by gradient descent The corresponding warp field range, thereby updating the main compensation event Then we maximize the event count image The density of main events on , refine the global activity area of ​​the subject event, and further adjust the subject compensation event It is worth noting that and Although both represent local measures of the deviation between subject and non-subject event regions in an event stream, they are based on different data sources: event rate and event timestamp.

[0238] Get the final event density of the main compensation event and regional scope After that, we can determine the range of events in the original event stream that can complement the main body in the new event stream. In this way, we further reduce the influence of non-main events with interfering effects in the new event stream, and complement the main events that may be deleted in step S3. Finally, the events in the time span of each spike data bin processed by the algorithm in this step are projected into the pixel plane in the form of an event count image, completing the representation processing flow of the entire event stream.

[0239] In summary, the present invention proposes a new and efficient dynamic asynchronous data optimization and representation method, which adaptively divides the event stream into segments of different lengths according to the spatiotemporal structural characteristics, correlation and polarity properties of the event stream itself; defines a new adaptive spatiotemporal subject surface compensation algorithm, which can complete the conversion of event stream data to frames with better performance and effect than the existing frame compression algorithm by compensating for the incomplete motion information of the event stream and deleting redundant information, and solves the shortcomings of the existing method when processing event streams with multiple subjects of different speeds coexisting.

[0240] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the disclosure of the present invention can be performed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in the present invention can be achieved, and this document does not limit this.

[0241] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A dynamic data optimization and representation method based on asynchronous event stream, It is characterized in that The specific steps include: S1. Divide the original event stream L of the event camera into the main event stream U and the non-main event stream N according to the amount of main information carried; S2. By establishing the complete energy function corresponding to the main event flow U and the original event flow L and the complete energy function corresponding to the non-main event flow N and the original event flow L, an event energy model based on Markov random field is constructed; after constructing the model, the main event flow is redefined as S i , define the non-subject event stream as S j ; The complete energy function corresponding to the main event flow U and the original event flow L The expression is as follows: ; in, The energy function expressions are: ; ; ; ; In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the main event stream U and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the main event stream U; α and β are non-negative dynamic weight parameters corresponding to the largest cluster; The complete energy function corresponding to the non-main event stream N and the original event stream L The expression is as follows: ; in, The energy function expressions are: ; ; ; ; In the above formula, Corresponding to the four largest groups , , , , the largest group It represents the relationship between the events in the non-subject event stream N and the events in the original event stream L. The three largest clusters represent the relationship between events in the spatiotemporal neighborhood of the non-subject event stream N; α and β are non-negative dynamic weight parameters corresponding to the largest cluster; S3. The main event stream S is converted into a time span according to the default time span. i and non-subject event stream S j Divide into several time bins; use dynamic asynchronous data measurement and slicing algorithm to adjust the length of the time bin span, then match the time bins one by one according to the sequence, and finally merge them into a new event stream; S4. According to the main event flow S i The correlation with the original event stream L is used to repair the part of the new event stream where the main information is located, and the redundant events in the time-space correlation area of ​​the new event stream are deleted to achieve information optimization; The dynamic asynchronous data measurement and slicing algorithm specifically includes: taking the polarity attribute of the event in the corresponding time bin in the event stream as the calculation basis of the overall density, and introducing the new event density into the time window as the basic reference basis, and dynamically adjusting the length of the time bin span; the adjustment of the time bin span is achieved by updating the time span, and the formula is as follows: In the formula, represents the time span of each time bin, and Represent the main event stream S i and non-subject event stream S j The time bin in and Represents two time bins and The time span before the update, and Represents two time bins respectively and The updated time span of and Represent the three-dimensional Gaussian kernel belonging to the i-th time bin and the j-th time bin respectively. , and represents the sum of the squared moduli of all time bins of the event stream, represents a space-time interval, Represents two time bins and The distance between them is given by the following formula: in, express and The inner product of represents the inner product of the ith time bin, represents the inner product of the j-th time bin.

2. According to claim 1, a method for optimizing and representing dynamic data based on asynchronous event streams, Features: The step S1 uses the main area recognition algorithm to divide the event stream; when performing area judgment and division operations on the event stream, the improved Pearson correlation coefficient is used and Spearman correlation coefficient , the formulas are as follows: ; ; in, Represents the horizontal axis coordinate of the i-th event; Represents the vertical axis coordinate of the i-th event; and It represents the weighted average of other events on the horizontal and vertical axes at the time node where the i-th event occurs, with the frequency of the event as the weight.

3. A method for optimizing and representing dynamic data based on asynchronous event stream according to claim 2, Features: The α reflects the spatial correlation of events, and the expression is as follows: ; The β reflects the temporal correlation of events, and the expression is as follows: ; In the above formula, Respectively represent the pixel difference between two events in the energy function in the horizontal and vertical directions, Respectively represent the maximum difference of events in all the largest clusters in the horizontal and vertical directions, and Respectively represent in all the largest groups The minimum and maximum time intervals between events.

4. A method for optimizing and representing dynamic data based on asynchronous event stream according to claim 3, It is characterized in that The step S4 of patching the part of the new event stream where the main information is located specifically adopts the following method: taking the local memory time plane of the new event stream as the basic operator, and grouping adjacent pixels into units of the same size and number; using the formula Calculate the initial warp field of each unit and update the warp field range by gradient descent to obtain the optimal unit compensation event density and regional scope The value of is used to determine the event range in the original event stream L that complements the main body in the new event stream and to repair it; In the formula, x represents the horizontal coordinate of the pixel corresponding to the event, y represents the vertical coordinate of the pixel corresponding to the event, E x and E y represents the local error in the plane displacement during the main event compensation process, E t Indicates the local error in the time axis during the main event compensation process. It represents the angle between the compensation event cloud after modeling with x, y, and t as the three axes and the z-axis represented by t, and t represents the time corresponding to the event.