Event feature extraction method based on spatial gradient variance maximization

Abstract

The invention relates to the technical field of image processing and computer vision, in particular to an event feature extraction method based on spatial gradient variance maximization, which comprises the following steps: acquiring event stream data shot by an event camera, processing the event stream data, and generating a binary event frame; determining an event Patch corresponding to each pixel in the binary event frame; obtaining a plurality of spatial gradients of each event Patch, and calculating a spatial gradient variance of the event Patch based on the plurality of spatial gradients; for the spatial gradient variances of all the events Patches, selecting the first m events Patches with the maximum spatial gradient variances to carry out de-overlapping processing so as to obtain a plurality of events Patches without any overlapping region, and taking the events Patches as a feature map of the binarized event frame; according to the invention, the accuracy and effectiveness of event camera data feature extraction can be improved.

Description

Technical Field

[0001] The invention relates to the technical field of image processing and computer vision, and in particular to an event feature extraction method based on maximizing spatial gradient variance. Background Art

[0002] In the field of image processing and computer vision, feature extraction is a key step to extract useful information from raw data. Traditional feature extraction methods, such as those based on edge detection, corner detection, or texture analysis, usually rely on static images or video frames. However, these methods often fail to provide sufficient real-time performance and accuracy when dealing with dynamic events, especially rapidly changing scenes. For example, in areas such as traffic monitoring, security surveillance, or robot vision, feature extraction techniques that can respond quickly and accurately identify events are required.

[0003] Existing event-based cameras can provide different data from traditional frame cameras. They only generate events when the brightness of pixels changes significantly, which makes them advantageous when dealing with dynamic scenes. However, how to extract useful features from these event data, especially while maintaining high temporal resolution, remains a challenge. Summary of the invention

[0004] In view of the above problems, the present invention provides an event feature extraction method based on maximizing spatial gradient variance, which solves the technical problem in the prior art of how to improve the accuracy and real-time performance of feature extraction of data captured by event cameras.

[0005] The present invention provides an event feature extraction method based on spatial gradient variance maximization, comprising the following steps: Step S1, acquiring event stream data captured by an event camera, processing the event stream data, and generating a binary event frame; Step S2, determining the event Patch corresponding to each pixel in the binary event frame; Step S3, obtaining multiple spatial gradients of each of the event patches, and calculating the spatial gradient variance of the event patch based on the multiple spatial gradients; Step S4: for the spatial gradient variances of all the event patches, select the first m event patches with the largest spatial gradient variances for de-overlapping processing to obtain multiple event patches without any overlapping areas as the feature map of the binary event frame.

[0006] Preferably, the step S1 specifically includes: Step S1-1, acquiring event stream data captured by an event camera, wherein the event stream data includes a plurality of events arranged in chronological order, and information of each event includes: pixel coordinates of the event, polarity of the event, and timestamp of the event; Step S1-2, dividing the events in the event stream data into multiple time windows according to the timestamps of the events; Step S1-3: For each of the time windows, generate a binary event frame based on the pixel coordinates of the event.

[0007] Preferably, the step S1-2 specifically includes: Step S1-2-1, obtaining the time range of the event stream data, including determining the start time and the end time; Step S1-2-2: Determine the time window , starting from the start time, based on the time window Dividing the time range into multiple time windows evenly; Step S1-2-3: For each event in the event stream data, assign the event to the corresponding time window according to the timestamp of the event.

[0008] Preferably, the step S1-3 specifically includes: Determine a binary event frame with width width and length height, and initialize all pixel values ​​to 255; Iterate over each event in each time window and locate the corresponding pixel position in the binary event frame according to the pixel coordinates of the event, regardless of the polarity of the event. Whether the value is positive or negative, the pixel value of the pixel position is set to 0, and the binary event frame is obtained at the coordinate The pixel value at , the expression is:

[0009] in, Represents the binary event frame at coordinates The pixel value at .

[0010] Preferably, the step S2 specifically includes: For each pixel in the binary event frame, an n×n pixel region is constructed with the pixel as the center and n as the side length. The pixel values ​​in the pixel region are the pixel values ​​at the corresponding positions in the binary event frame. The pixel region is determined as the event Patch corresponding to the pixel.

[0011] Preferably, the step S3 specifically includes: Step S3-1: For each event Patch, the horizontal gradient and the vertical gradient of each pixel in the event Patch are calculated, and the spatial gradient of each pixel is determined by the horizontal gradient and the vertical gradient of each pixel, and the expression is:

[0012] in, Indicates the first The spatial gradient of pixels, and Respectively represent The horizontal and vertical gradients of pixels; Step S3-2: Calculate the variance of the spatial gradients of all pixels in the event patch as the spatial gradient variance of the event patch. The expression is:

[0013] in, Represents the total number of pixels in the event Patch. Represents the spatial gradient variance of the event Patch.

[0014] Preferably, the step S4 specifically includes: Step S4-1, sort the spatial gradient variances of all event patches, and select the first m event patches with the largest spatial gradient variances as the candidate event patch set; Step S4-2, performing de-overlapping processing on the candidate event Patch set to obtain a feature map of a binary event frame, specifically comprising: Step S4-2-1, determine the minimum distance threshold min_distance, and initialize the non-overlapping event Patch set to an empty set; Step S4-2-2, selecting all event patch pairs from the candidate event patch set by enumerating all combinations; for each event patch pair, calculating the Manhattan distance between the two event patches in the event patch pair, and if the Manhattan distance is greater than the minimum distance threshold min_distance, adding the event patch pair to the non-overlapping event patch set; Step S4-2-3: determine all event patches in the final non-overlapping event patch set as the multiple event patches without any overlapping areas, and use them as the feature map of the binary event frame.

[0015] Compared with the prior art, the present invention has at least the following beneficial effects: (1) The present invention converts the event stream data captured by the event camera into binary event frames and ignores the polarity of the event, while representing both positive and negative polarity events. This allows the present invention to retain key feature information, better adapt to different types of event changes, and improve the stability and accuracy of feature extraction.

[0016] (2) The present invention can effectively quantify and evaluate the information richness within each patch by calculating the spatial gradient variance of the event patch. Larger variance indicates areas with drastic gradient changes, which usually contain more image details and feature information. By selecting these high-variance areas as feature maps, the system can more accurately capture key details in the image, providing richer and more meaningful feature representations for subsequent recognition, classification, and tracking.

[0017] (3) In the process of generating feature maps, the present invention ensures that there is no overlap between the selected event patches through de-overlapping processing. This not only improves the clarity of the feature map, but also avoids information redundancy, thereby making the feature representation more concise and efficient. The resulting non-overlapping feature map provides a more accurate basis for subsequent event analysis and processing, and improves the performance and reliability of the overall system. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The drawings are only for the purpose of illustrating particular embodiments and are not to be construed as limiting the invention.

[0019] Figure 1 A flow chart of the event feature extraction method based on spatial gradient variance maximization provided by the present invention.

[0020] Figure 2 This is a schematic diagram of a binarized event frame provided by the present invention.

[0021] Figure 3 A schematic diagram of marking a feature map in a binarized event frame provided by the present invention. DETAILED DESCRIPTION

[0022] In order to more clearly understand the above-mentioned purpose, features and advantages of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings and specific embodiments. It should be noted that, in the absence of conflict, the embodiments of the present invention and the features in the embodiments can be combined with each other. In addition, the present invention can also be implemented in other ways different from those described herein, and therefore, the protection scope of the present invention is not limited by the specific embodiments disclosed below.

[0023] The present invention provides an event feature extraction method based on maximizing spatial gradient variance, which extracts key event features by analyzing event triggering data at the pixel level to facilitate subsequent event recognition, classification and tracking.

[0024] In order to illustrate the effectiveness of the method proposed by the present invention, the above technical solution of the present invention is described in detail below through a specific embodiment. Figure 1 As shown, a method for extracting event features based on maximizing spatial gradient variance is disclosed, and the specific implementation steps are as follows: Step S1: Acquire event stream data captured by an event camera, process the event stream data, and generate a binary event frame.

[0025] Unlike traditional cameras, event cameras capture brightness changes in a scene. They generate event stream data by photographing the scene. Event stream data is a data format used to record dynamic visual information. Event stream data is in the form of an event sequence, which contains multiple events arranged in chronological order. Each event contains Parameters. Among them, is the pixel coordinate of the event, indicating the location where the event occurs in the image. is the polarity of the event, indicating the direction of brightness change, A positive value of indicates an increase in brightness. Negative values ​​indicate decreased brightness. The timestamp of the event, indicating the time when the event occurred.

[0026] In some embodiments, the event stream data can be denoised by applying a noise filtering algorithm, such as an event density-based threshold filtering algorithm or a time neighborhood consistency algorithm, to remove isolated or inconsistent events in the event stream data, which helps to improve the accuracy of the generated binary event frames and ensure the reliability of subsequent feature extraction.

[0027] The present invention adopts a segmented processing method to divide the events in the event stream data into multiple time windows according to timestamps. The specific steps include: Get the time range of event stream data, including determining the start time and end time; Determine the time window , starting from the start time, based on the time window Dividing the time range into multiple time windows evenly; For each event in the event stream data, the event is assigned to the corresponding time window according to the timestamp of the event.

[0028] The present invention generates a binary event frame for each time window based on the pixel coordinates of the event. The specific steps include: Determine a binary event frame with a size of width and height, and initialize all pixel values ​​to 255 (white), indicating that there is no event in the initial state; Traverse each event in the current time window, locate the corresponding pixel position in the binary event frame according to the pixel coordinates of the event, and set the pixel value of the pixel position to 0 (black), indicating that an event has occurred; specifically, in the present invention, regardless of the polarity of the event The value of is positive or negative. Whenever an event occurs, the pixel value is set to 0 (black). The specific expression is:

[0029] in, Represents the binary event frame at coordinates The pixel value at .

[0030] like Figure 2 The figure shows the binarized event frame generated by the present invention. Through this step, the present invention converts the event stream data captured by the event camera into a visualized binarized event frame, ignoring the polarity of the event during the processing, and can simultaneously represent positive and negative polarity events, which is convenient for subsequent event feature extraction, analysis and visualization.

[0031] Step S2: Determine the event Patch corresponding to each pixel in the binary event frame.

[0032] The event patch of the present invention specifically represents a pixel area of ​​size n×n. For each pixel in the binary event frame, an n×n pixel area is formed with the pixel as the center and n as the side length. The pixel value in the pixel area is the pixel value of the corresponding position in the binary event frame, and the pixel area is determined as the event patch corresponding to the pixel. By establishing an event patch, the present invention can be used to analyze the distribution and characteristics of events in a local range.

[0033] Step S3: Acquire multiple spatial gradients of each of the event patches, and calculate the spatial gradient variance of the event patch based on the multiple spatial gradients.

[0034] The present invention adopts a sliding window method to slide pixel by pixel to traverse all event patches. The processing steps for each event patch are as follows: For the first pixels, calculate the horizontal gradient and the vertical gradient .

[0035] In some embodiments, a gradient operator such as a Sobel operator, a Scharr operator, or a Laplacian operator may be used to calculate the gradients in the horizontal and vertical directions.

[0036] The square sum of the horizontal and vertical gradients determines the The spatial gradient of pixels is expressed as:

[0037] in, Indicates the first The spatial gradient of pixels.

[0038] After obtaining the spatial gradients of all pixels in the event patch, the variance of the spatial gradients of all pixels in the event patch is calculated as the spatial gradient variance of the event patch. The expression is:

[0039] in, Represents the total number of pixels in the event Patch. Represents the spatial gradient variance of the event Patch.

[0040] In some embodiments, an integral graph optimization method may be used to optimize the calculation of the variance.

[0041] In some embodiments, the present invention determines the variance kernel of the event patch by the spatial gradient variance of the event patch. . Variance Kernel It is used to represent the whole process of calculating the spatial gradient variance of the event patch. The variance is used to check the event frame Performing a convolution operation means calculating the spatial gradient variance of all event patches by sliding the window pixel by pixel.

[0042] The spatial gradient variance of the present invention reflects the degree of discreteness of the gradient within the event patch. A larger variance indicates that the gradient within the event patch changes dramatically, and usually contains more image details and feature information. By calculating the spatial gradient variance, the information richness within the event patch can be quantitatively evaluated.

[0043] Step S4: for the spatial gradient variances of all the event patches, select the first m event patches with the largest spatial gradient variances for de-overlapping processing to obtain multiple event patches without any overlapping areas as the feature map of the binary event frame.

[0044] In this step, the spatial gradient variances of all event patches are sorted, and the first m event patches with the largest spatial gradient variances are selected as the candidate event patch set; The candidate event Patch set is subjected to de-overlapping processing, specifically comprising: Determine the minimum distance threshold min_distance and initialize the non-overlapping event Patch set to an empty set; Obtain all event patch pairs from the candidate event patch set by enumerating all combinations, calculate the Manhattan distance between two event patches in the event patch pair, and if the Manhattan distance is greater than the minimum distance threshold min_distance, add the event patch pair to the non-overlapping event patch set; After traversing all event patch pairs, all event patches in the non-overlapping event patch set are determined as the multiple event patches without any overlapping areas, and used as the feature map of the binarized event frame.

[0045] In some embodiments, Figure 3 As shown, the feature map of the binary event frame is annotated in the original binary event frame, and a rectangular box is drawn for each event patch without any overlapping area, and then Gaussian blur and pseudo color processing are performed to enhance the visualization effect.

[0046] Through this step, the area with the greatest spatial variation is selected, and multiple event patches without any overlapping areas are generated as feature maps, thereby providing effective feature representation for subsequent event recognition, classification, and tracking methods.

[0047] Although the specific embodiments of the present invention depict various actions or steps in a specific order, this should be understood as requiring such actions or steps to be performed in the specific order shown or in a sequential order, or requiring all illustrated actions or steps to be performed to obtain the desired results. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although several specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Certain features described in the context of a separate embodiment can also be implemented in a single implementation in combination. On the contrary, the various features described in the context of a single implementation can also be implemented in multiple implementations individually or in any suitable sub-combination. The above is only a preferred specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or replacements that can be easily thought of by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

[0048] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. An event feature extraction method based on spatial gradient variance maximization, characterized in that: The following steps are involved: Step S1, acquiring event stream data captured by an event camera, processing the event stream data, and generating a binary event frame; Step S2, determining the event Patch corresponding to each pixel in the binary event frame; Step S3, obtaining multiple spatial gradients of each of the event patches, and calculating the spatial gradient variance of the event patch based on the multiple spatial gradients; Step S4: for the spatial gradient variances of all the event patches, select the first m event patches with the largest spatial gradient variances for de-overlapping processing to obtain multiple event patches without any overlapping areas as the feature map of the binary event frame.

2. The event feature extraction method based on spatial gradient variance maximization according to claim 1 is characterized in that: The step S1 specifically includes: Step S1-1, acquiring event stream data captured by an event camera, wherein the event stream data includes a plurality of events arranged in chronological order, and information of each event includes: pixel coordinates of the event, polarity of the event, and timestamp of the event; Step S1-2, dividing the events in the event stream data into multiple time windows according to the timestamps of the events; Step S1-3: For each of the time windows, generate a binary event frame based on the pixel coordinates of the event.

3. The event feature extraction method based on spatial gradient variance maximization according to claim 2 is characterized in that: The step S1-2 specifically includes: Step S1-2-1, obtaining the time range of the event stream data, including determining the start time and the end time; Step S1-2-2: Determine the time window , starting from the start time, based on the time window Dividing the time range into multiple time windows evenly; Step S1-2-3: For each event in the event stream data, assign the event to the corresponding time window according to the timestamp of the event.

4. The event feature extraction method based on spatial gradient variance maximization according to claim 3 is characterized in that: The step S1-3 specifically includes: Determine a binary event frame with width width and length height, and initialize all pixel values ​​to 255; Traverse each event in each time window and locate the corresponding pixel position in the binary event frame according to the pixel coordinates of the event, regardless of the polarity of the event Whether the value is positive or negative, the pixel value of the pixel position is set to 0, and finally the binary event frame is obtained at the coordinate The pixel value at , the expression is: in, Represents the binary event frame at coordinates The pixel value at .

5. The event feature extraction method based on spatial gradient variance maximization according to claim 4 is characterized in that: The step S2 specifically includes: For each pixel in the binary event frame, an n×n pixel region is constructed with the pixel as the center and n as the side length. The pixel values ​​in the pixel region are the pixel values ​​at the corresponding positions in the binary event frame. The pixel region is determined as the event Patch corresponding to the pixel.

6. The event feature extraction method based on spatial gradient variance maximization according to claim 5 is characterized in that: The step S3 specifically includes: Step S3-1: For each event Patch, the horizontal gradient and the vertical gradient of each pixel in the event Patch are calculated, and the spatial gradient of each pixel is determined by the horizontal gradient and the vertical gradient of each pixel, and the expression is: in, Indicates the first The spatial gradient of pixels, and Respectively represent The horizontal and vertical gradients of pixels; Step S3-2: Calculate the variance of the spatial gradients of all pixels in the event patch as the spatial gradient variance of the event patch. The expression is: in, Represents the total number of pixels in the event Patch. Represents the spatial gradient variance of the event Patch.

7. The event feature extraction method based on spatial gradient variance maximization according to claim 6 is characterized in that: The step S4 specifically includes: Step S4-1, sort the spatial gradient variances of all event patches, and select the first m event patches with the largest spatial gradient variances as the candidate event patch set; Step S4-2, performing de-overlapping processing on the candidate event Patch set to obtain a feature map of a binary event frame, specifically comprising: Step S4-2-1, determine the minimum distance threshold min_distance, and initialize the non-overlapping event Patch set to an empty set; Step S4-2-2, selecting all event patch pairs from the candidate event patch set by enumerating all combinations; for each event patch pair, calculating the Manhattan distance between the two event patches in the event patch pair, and if the Manhattan distance is greater than the minimum distance threshold min_distance, adding the event patch pair to the non-overlapping event patch set; Step S4-2-3: determine all event patches in the final non-overlapping event patch set as the multiple event patches without any overlapping areas, and use them as the feature map of the binary event frame.

Images