An event information visualization method with adaptive time resolution

Abstract

The present invention relates to a method for visualizing event information with adaptive time resolution, including: Step 1, set an initial time interval, and project the events within the initial time interval onto two all-zero matrices respectively according to the positive and negative of the event polarity. In each matrix, assign 1 to the positions where events exist to obtain two decomposed diagrams of events; Step 2, perform median filtering processing and RGB conversion on the two decomposed diagrams respectively to obtain an initial diagram, and count the number of all overlapping points in the initial diagram; Step 3, determine whether the number of overlapping points is within the determination range. If so, use the initial diagram as the generated diagram for visualizing event information; if not, adjust the initial time interval and return to Step 1. The present invention can adaptively change the time resolution, ensure that the visualization of events is always completed with clear and relatively minimal number of images, is easier to capture the details of object movement, and can save data volume.

Description

Technical Field

[0001] The present invention relates to the technical field of spatial intelligent imaging, and particularly to an event information visualization method with adaptive time resolution. Background Art

[0002] With the rapid development of spatial imaging technology, people's demand for more information in a single image is becoming more and more urgent. While meeting the high information volume, people's demand for low data volume and high timeliness of spatial imaging is also increasing.

[0003] Traditional CMOS or CCD sensors can obtain a large amount of information when imaging the ground, but the price is that they will generate an extremely high amount of data, bringing a huge burden to data transmission and processing, and seriously affecting timeliness. To address this contradiction, many researchers have tried to reduce the data volume during data transmission through methods such as data compression, but this will increase the time for subsequent data processing and still cannot effectively resolve this contradiction.

[0004] In recent years, some people have proposed applying a Dynamic Vision Sensor (DVS) to spatial imaging to resolve the above contradiction. A single pixel of the dynamic vision sensor can detect the change in light intensity in the scene in real time and output event information containing only a timestamp, pixel position, and event polarity when the change degree exceeds a certain threshold. The unique working principle and data output form of the dynamic vision sensor enable it to reduce the imaging data volume from the source, and its feature of microsecond-level time resolution also ensures the demand for a large amount of information. However, the higher the time resolution, the greater the total number of events, and different scenarios have different requirements for time resolution. At the same time, due to the independence of event information and the particularity of its information structure, the visualization degree of event information is low and it is impossible to conveniently complete information processing. Therefore, there is an urgent need for an event information visualization method that can reasonably adjust the time resolution of the dynamic vision sensor according to different shooting scenarios. Summary of the Invention

[0005] The present invention provides an event information visualization method with adaptive time resolution to solve the problems of low visualization degree of event information and different requirements for time resolution in different scenarios.

[0006] To achieve the above object, the present invention adopts the following technical solutions:

[0007] An event information visualization method with adaptive time resolution, comprising the following steps:

[0008] Step 1: Set an initial time interval. According to the event information, project the events within the initial time interval onto two all-zero matrices with the same spatial resolution as the dynamic vision sensor, respectively, based on the positive and negative polarities of the events. Assign the value 1 to the positions where events exist in each matrix to obtain two decomposed diagrams of events.

[0009] Step 2: Perform median filtering on the two decomposed diagrams respectively, and then convert the two decomposed diagrams to RGB to obtain an initial image. During the conversion, if there are both positive-polarity and negative-polarity events at the same pixel position, mark this pixel position as an overlapping point, and count the number of all overlapping points in the initial image.

[0010] Step 3: Determine whether the number of overlapping points is within the determination range. If so, use the initial image as the generated image for visualizing event information; if not, continue to determine whether the number of overlapping points is greater than or equal to or less than or equal to the determination range, and accordingly reduce or increase the initial time interval by the minimum time unit and return to Step 1.

[0011] The beneficial effects of the present invention are as follows:

[0012] The method for visualizing event information with adaptive time resolution proposed by the present invention converts the invisible event information obtained from a dynamic vision sensor into image information, so that various advanced algorithms for processing images can be used to analyze and process the event information. In addition, the images obtained from the events are no longer presented at a fixed frame rate, but are adaptively changed according to the speed of object movement or the change speed of light intensity in the scene to present the time resolution, which can ensure that the visualization of events is always completed with clear and relatively minimal number of images, making it easier to capture the details of object movement and saving data volume, bringing speed and convenience to subsequent analysis and processing work. Description of the Drawings

[0013] Figure 1 It is a flowchart of the method for visualizing event information with adaptive time resolution in an embodiment of the present invention. Detailed Embodiments

[0014] In order to make the technical solutions of the present application clearer, the technical solutions of the present invention will be further specifically described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0015] This embodiment provides a method for visualizing event information with adaptive time resolution, as Figure 1 shown. The method includes the following steps:

[0016] Step 1: Set the initial time interval. According to the event information, project the events within the initial time interval onto two all-zero matrices with the same resolution as the dynamic vision sensor, respectively, according to the positive and negative of the event polarity. And in each matrix, assign the value 1 to the positions where events exist. After the assignment, two decomposed diagrams of events are obtained.

[0017] In this step, the event information mainly comes from the dynamic vision sensor, specifically including: the time stamp when the event occurs, the coordinate values of the position where the event is located, and the polarity of the event. The polarity of the event specifically includes: positive polarity when the brightness increases beyond the threshold, and negative polarity when the brightness decreases beyond the threshold.

[0018] The dynamic vision sensor is a sensor that only focuses on the change of light intensity. It will detect the light intensity at the corresponding pixel position in real time, and when the light intensity change exceeds the threshold, it will output an event. This event records the time t when this change occurs, the pixel position coordinates (x, y) where the change occurs, and whether it brightens or darkens p. The value of p is replaced by +1 and -1, where +1 represents brightening and -1 represents darkening.

[0019] For example, in this embodiment, the events are collected from DAVIS346. The spatial resolution of the collected events is 260*346, and the time resolution is 1 microsecond. Set the initial time interval Tg, such as Tg = 20000 microseconds. According to the value of time t in the event information, all events within the time period [t, t+Tg] can be screened out, and then these events are projected onto two all-zero matrices of 260*346 according to the coordinates (x, y) of the screened events.

[0020] During the projection, according to the positive and negative of the polarity p of each event, project the events with positive polarity onto one all-zero matrix, and project the events with negative polarity onto another all-zero matrix. And in each all-zero matrix, assign the value 1 to the positions where events exist, and the values of other positions are 0. After the assignment of all positions, two decomposed diagrams of events can be obtained. At the same time, for the sake of distinction, use green to represent the positions containing positive-polarity events in the all-zero matrix, and use red to represent the positions containing negative-polarity events in the all-zero matrix.

[0021] In the projection process, there may be multiple events at the same pixel position. In this case, only the last-occurring event needs to be retained, and the previous events are discarded.

[0022] Through the above steps, a series of invisible event information can be converted into visible image information respectively containing brightening and darkening information.

[0023] Step 2: Perform median filtering on the two decomposed images respectively, and then perform RGB conversion on the two decomposed images to obtain an initial image. During the conversion, if there are both positive-polarity events and negative-polarity events at the same pixel position, mark this pixel position as an overlapping point, and count the number of all overlapping points in the initial image.

[0024] Due to the continuity of moving objects, the actually generated events should be correlated, that is, there should also be other real events near a certain real event in a short period of time. According to this characteristic of real events, sporadic noise events in the image can be removed. Most of these events belong to thermal noise. At the same time, holes occasionally generated due to unstable imaging and other reasons in the real event array can be filled, which can make the interior of the object more continuous, the edges smoother, and enhance the information expression ability.

[0025] According to the characteristic that the actually occurring real events are correlated, perform median filtering on the two decomposed images respectively to remove the scattered noise events in the image and fill the holes in the object pixel array, so as to achieve the purpose of enhancing image information.

[0026] Specifically, when performing median filtering on the two decomposed images respectively, use a median filter with a size of 3*3 to perform median filtering on the two decomposed images respectively. Statistically sort the values of the target pixel position and its surrounding 8 adjacent pixel positions and determine the median of the sorting. If the value of the target pixel position is less than the median, assign the value 0 to this target pixel position; if the value of the target pixel position is greater than or equal to the median, assign the value 1 to this target pixel position. After traversing all pixel positions in the two decomposed images, the median filtering of the decomposed images is completed.

[0027] The two denoised decomposed images respectively contain information about brightening and darkening. Perform RGB conversion on the two decomposed images to obtain an initial image. During the conversion, if there are both brightening events (i.e., positive-polarity events) and darkening events (i.e., negative-polarity events) at the same pixel position, mark this pixel position as an overlapping point, and count the number of all overlapping points in the initial image, denoted as W. At the same time, for the convenience of distinction, use yellow to represent the pixel positions where the overlapping points are located in the initial image.

[0028] The significance of the overlapping point is that within the time period [t, t+Tg], the brightness at this pixel position has both brightened and darkened, indicating that the brightness change at this pixel position is relatively fast. In the real world when the lighting conditions remain unchanged, it means that the object at this pixel position moves relatively fast.

[0029] Step 3: Determine whether the number of overlapping points is within the determination range. If it is, use the initial graph as the generated graph for visualizing event information; if not, continue to determine whether the number of overlapping points is greater than or equal to, or less than or equal to, the determination range, and accordingly reduce or increase the initial time interval by the minimum time unit and then return to Step 1.

[0030] For an initial graph, when the lighting conditions remain unchanged, the number of overlapping points it contains indirectly reflects the speed of the object being photographed. If the object moves too fast, a large number of overlapping points will be generated during the time period [t, t+Tg], making it impossible to distinguish the features of the object in the initial graph; on the contrary, if the object moves too slowly, there will be too few brightening and darkening events during the time period [t, t+Tg], making it impossible to accurately present the features of the object in these events in the initial graph. Therefore, the number of overlapping points can be used to define the time resolution of the initial graph.

[0031] In this step, a determination range for the number of overlapping points needs to be set. If the number of overlapping points is higher than this range, it indicates that the overlapping phenomenon of brightening and darkening events is relatively serious. At this time, the object being photographed moves at a relatively high speed, resulting in crowded events, that is, the initial time interval is too large and should be appropriately reduced. On the contrary, if it is lower than this range, it indicates that there is almost no overlapping phenomenon of brightening and darkening events. At this time, the object being photographed moves at a relatively low speed, resulting in too sparse events, that is, the initial time interval is small and should be appropriately increased.

[0032] Specifically, set the determination range to (10, 20); and then specify a minimum time unit d according to the photographed environment. For example, in this embodiment, d = 100 microseconds is specified.

[0033] Determine the size relationship between the number of overlapping points W and the determination range Z±a: If W≥Z+a, it indicates that the overlapping phenomenon of brightening and darkening events is relatively serious, the object in the graph moves relatively fast, and the events are crowded. Then discard this initial graph, reduce the initial time interval Tg by the minimum time unit d, that is, update the initial time interval to Tg - d, and then return to Step 1; if W≤Z - a, it indicates that there are no overlapping points in the initial graph, the object in the graph moves relatively slowly, and the number of brightening and darkening events is small. Then discard this initial graph, increase the initial time interval Tg by the minimum time unit d, that is, update the initial time interval to Tg + d, and then return to Step 1; if Z - a < W < Z + a, it indicates that the event occupancy ratio is relatively reasonable. Then retain this initial graph as the generated graph with clear object features to complete the visualization of event information.

[0034] The proposed method for visualizing event information with adaptive time resolution in the present invention converts the invisible event information obtained from a dynamic vision sensor into image information, so that various advanced algorithms for processing images can be used to analyze and process the event information. In addition, the images obtained from the events are not presented at a fixed frame rate, but are adaptively changed in time resolution according to the speed of object movement or the change speed of light intensity in the scene, which can ensure that the visualization of the event is always completed with clear and relatively minimal number of images, making it easier to capture the details of object movement and saving data volume, bringing speed and convenience to subsequent analysis and processing work.

[0035] The technical features of the above-described embodiments can be combined arbitrarily. For the sake of concise description, not all possible combinations of the technical features in the above-described embodiments are described. However, as long as there is no contradiction in the combination of these technical features, it should be considered as the scope described in this specification.

[0036] The above-described embodiments only represent several implementation manners of the present invention, and the description thereof is relatively specific and detailed, but it should not be construed as a limitation on the scope of the invention patent. It should be noted that for those of ordinary skill in the art, without departing from the concept of the present invention, several deformations and improvements can be made, and these all belong to the protection scope of the present invention. Therefore, the protection scope of the present invention patent shall be subject to the appended claims.

Claims

1. An event information visualization method with adaptive time resolution, characterized in that, It includes the following steps: Set an initial time interval; Step 1: According to the event information, project the events within the initial time interval onto two all-zero matrices with the same spatial resolution as the dynamic vision sensor, respectively, according to the positive and negative polarities of the event polarities. Assign the value 1 to the positions where events exist in each matrix to obtain two decomposed diagrams of events. The event information includes the time stamp when the event occurs, the coordinate values of the position where the event is located, and the polarity of the event. The polarity of the event specifically includes: positive polarity when the brightness increases by more than the threshold, and negative polarity when the brightness decreases by more than the threshold; Step 2: Perform median filtering on the two decomposed diagrams respectively, and then perform RGB conversion on the two decomposed diagrams to obtain an initial diagram. When converting, if there are both positive-polarity and negative-polarity events at the same pixel position, mark this pixel position as an overlapping point, and count the number of all overlapping points in the initial diagram; Step 3: Determine whether the number of overlapping points is within the determination range. If so, use the initial diagram as the generated diagram for event information visualization; if not, continue to determine whether the number of overlapping points is greater than or equal to or less than or equal to the determination range. When the number of overlapping points is greater than or equal to the determination range, reduce the initial time interval by the minimum time unit and return to Step 1. When the number of overlapping points is less than or equal to the determination range, increase the initial time interval by the minimum time unit and return to Step 1.

2. The event information visualization method with adaptive time resolution according to claim 1, wherein When performing median filtering on the two decomposed diagrams respectively, it includes the following steps: Use a median filter with a size of 3*3 to perform median filtering on the two decomposed diagrams respectively. Statistically sort the values of the target pixel position and its surrounding 8 adjacent pixel positions and determine the median of the sorting. If the value of the target pixel position is less than the median, assign 0 to this target pixel position; if the gray value of the target pixel position is greater than or equal to the median, assign 1 to this target pixel position; After traversing all pixel positions in the two decomposed diagrams, complete the median filtering process.

3. The method for visualizing event information with adaptive time resolution according to claim 1 or 2, characterized in that, In the two all-zero matrices, the positions containing positive-polarity events are represented in green, and the positions containing negative-polarity events are represented in red; in the initial diagram, the pixel positions where the overlapping points are located are represented in yellow.

4. The method for visualizing event information with adaptive time resolution according to claim 1 or 2, characterized in that The initial time interval is 20,000 microseconds, and the minimum time unit is 100 microseconds.

5. The method for visualizing event information with adaptive time resolution according to claim 1 or 2, characterized in that, The determination range is (10, 20).

Images