Event data tagging methods and related equipment based on neuromorphic vision

Abstract

This invention relates to the field of flow cytometry cell sorting technology, specifically disclosing an event data labeling method and related equipment based on neuromorphic vision. The method includes the following steps: acquiring an event data stream to be processed, and segmenting the event data stream based on a pre-defined time window to generate a set of window events to be processed; performing noise filtering on each window event set, and retaining window events with a remaining number greater than a preset first threshold to form a valid event set; extracting key point information for each valid event set based on unsupervised clustering and / or feature detection, according to preset size parameters; performing trajectory tracking on the key point information and configuring trajectory IDs to generate trajectory information for labeling the event data stream. This method can fully utilize the high temporal accuracy of event cameras to achieve cell labeling, avoiding the temporal resolution loss caused by traditional image-assisted labeling methods.

Description

Technical Field

[0001] This application relates to the field of flow cytometry sorting technology, and more specifically, to an event data labeling method and related equipment based on neuromorphic vision. Background Technology

[0002] In the field of imaging flow cytometry cell sorting, neuromorphic visual event cameras, with their high temporal resolution and low data rate, are gradually replacing traditional machine vision cameras, providing a new technological path for the rapid perception and real-time sorting of high-speed flowing cells. The application of event data shows broad prospects, especially in creating efficient datasets, laying the foundation for the training of subsequent sorting and classification models.

[0003] However, existing methods for event data labeling primarily rely on image-based methods, such as converting event data into a two-dimensional grid to generate an image representation followed by manual annotation, or using high-speed camera images for labeling assistance. This labeling approach significantly reduces the temporal resolution of events, failing to fully utilize the high temporal accuracy of event cameras, resulting in labels that cannot accurately match microsecond-level changes in events. Furthermore, this method struggles to precisely distinguish between target object events and noisy events, leading to inaccurate and easily interfered labeling results. The labeling process is cumbersome, requires significant manual intervention, and is inefficient and costly.

[0004] There is currently no effective technical solution to the above problems. Summary of the Invention

[0005] The purpose of this application is to provide an event data labeling method and related equipment based on neuromorphic vision, so as to realize the automatic labeling of event data, make full use of the high temporal accuracy of event cameras, reduce labor costs, and improve labeling efficiency.

[0006] In a first aspect, this application provides an event data labeling method based on neuromorphic vision for labeling cellular events acquired using an event camera. The method includes the following steps:

[0007] S1. Obtain the event data stream to be processed, and divide the event data stream according to a pre-defined time window to generate a set of window events to be processed;

[0008] S2. Perform noise filtering on each window event set and retain window events with a remaining number greater than a preset first threshold to form a valid event set.

[0009] S3. Based on unsupervised clustering and / or feature detection, extract key point information of each effective event set according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size;

[0010] S4. Track the key point information and configure the trajectory ID to generate trajectory information to mark the event data stream.

[0011] This application's method fully leverages the high temporal accuracy of event cameras for cell labeling, avoiding the temporal resolution loss inherent in traditional image-assisted labeling methods. Through noise filtering, it effectively removes noise from event data, improving the purity of the labeling results. Then, based on unsupervised clustering and / or feature detection, as well as keypoint extraction of size parameters, it can more accurately identify and locate cell events, distinguishing between target events and noisy events. The trajectory tracking process provides continuous motion information of cell events. The entire method achieves high-precision and high-efficiency labeling of cell events, reducing manual intervention and improving data processing efficiency.

[0012] In the aforementioned event data labeling method based on neuromorphic vision, in step S2, noise filtering is performed based on one or more cascaded event noise filters, wherein the event noise filters include background noise filters and refractory period filters.

[0013] In this example, the proposed method can effectively remove noise from different sources in event data, including background noise and noise caused by the refractory period effect, improving the thoroughness of noise filtering and thus improving the quality of the effective event set. This provides a cleaner data foundation for subsequent key point extraction and trajectory tracking, and improves the accuracy and reliability of the entire event data labeling method.

[0014] The event data labeling method based on neuromorphic vision, wherein in step S3, the size parameter includes distance boundary, and the process of extracting key point information of each valid event set based on unsupervised clustering according to the preset size parameter includes:

[0015] S31. Cluster the events in the effective event set based on the unsupervised clustering algorithm to obtain clustering results containing several event clusters;

[0016] S32. Filter out event clusters in each clustering result whose number of events is less than a preset second threshold;

[0017] S33. Filter outlier events from the remaining event clusters based on the distance boundary;

[0018] S34. Output key information based on the event clusters after filtering outlier events.

[0019] The event data labeling method based on neuromorphic vision, wherein in step S3, the size parameters include the minimum circle radius, the maximum circle radius, and the minimum circle center distance, and the process of extracting key point information of each valid event set based on feature detection and according to the preset size parameters includes:

[0020] S31' Convert the events in each valid event set into a two-dimensional edge graph and obtain the corresponding event timestamps;

[0021] S32', Based on the minimum circle radius, maximum circle radius, and minimum circle center distance, examine the edge map using Hough circle detection to label cell events based on circles;

[0022] S33' Outputs the key point information based on circular cell events and the event timestamps.

[0023] The event data labeling method based on neuromorphic vision, wherein the step between step S31' and step S32' further includes the following step:

[0024] S3A', Perform downsampling processing on the edge map.

[0025] The event data labeling method based on neuromorphic vision, wherein step S4 includes:

[0026] S41. Group the key point information according to the pixel distance and time difference between key point information, and assign trajectory IDs to obtain preliminary trajectory information;

[0027] S42. For each preliminary trajectory information, merge the key point information whose time difference is less than a preset first time threshold to obtain the trajectory information to mark the event data stream.

[0028] The event data labeling method based on neuromorphic vision, wherein the calibration process of the time window and the reference size includes:

[0029] A1. Acquire the event data stream for testing, and mark the event data stream for testing based on a preset initial time window and initial reference size to obtain pre-detection trajectory information;

[0030] A2. Calculate the average velocity information of cell events based on the pre-detection trajectory information;

[0031] A3. Calculate and calibrate the time window based on the ratio of the expected pixel movement speed of the set cell event to the average speed information;

[0032] A4. Calculate the average size information corresponding to the cell events based on the pre-detection trajectory information, and use it as the reference size.

[0033] Secondly, this application also provides an event data labeling system based on neuromorphic vision for labeling cellular events acquired using an event camera, the system comprising:

[0034] The preprocessing module is used to acquire the event data stream to be processed, and to divide the event data stream according to a pre-calibrated time window to generate a set of window events to be processed. It is also used to perform noise filtering on each set of window events and to retain window events with a remaining number of events greater than a preset first number threshold to form a valid event set.

[0035] The key point detection module is used to extract key point information of each effective event set based on unsupervised clustering and / or feature detection, according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size;

[0036] The key point tracking module is used to track the trajectory of the key point information, configure the trajectory ID, and generate trajectory information to mark the event data stream.

[0037] This system fully leverages the high temporal accuracy of event cameras for cell labeling, avoiding the temporal resolution loss inherent in traditional image-assisted labeling methods. Through noise filtering, it effectively removes noise from event data, improving the purity of the labeling results. Then, based on unsupervised clustering and / or feature detection, as well as keypoint extraction of size parameters, it can more accurately identify and locate cell events, distinguishing between target events and noisy events. The trajectory tracking process provides continuous motion information of cell events. The entire system achieves high-precision and high-efficiency labeling of cell events, reducing manual intervention and improving data processing efficiency.

[0038] Thirdly, this application also provides an electronic device, including a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method provided in the first aspect above.

[0039] Fourthly, this application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the method provided in the first aspect above.

[0040] As described above, this application provides an event data labeling method and related equipment based on neuromorphic vision. The method fully utilizes the high temporal accuracy of event cameras to achieve cell labeling, avoiding the temporal resolution loss associated with traditional image-assisted labeling methods. Through noise filtering, it effectively removes noise from the event data, improving the purity of the labeling results. Then, based on unsupervised clustering and / or feature detection, as well as key point extraction of size parameters, it can more accurately identify and locate cell events, distinguishing between target events and noise events. The trajectory tracking process provides continuous motion information of cell events. The entire method achieves high-precision and high-efficiency labeling of cell events, reducing manual intervention and improving data processing efficiency. Attached Figure Description

[0041] Figure 1 A flowchart of an event data tagging method based on neuromorphic vision provided in an embodiment of this application.

[0042] Figure 2 A detailed flowchart illustrating the event data labeling method based on neuromorphic vision provided in this application embodiment.

[0043] Figure 3 This is a schematic diagram of the structure of an event data labeling system based on neuromorphic vision provided in an embodiment of this application.

[0044] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0045] Reference numerals: 201, preprocessing module; 202, key point detection module; 203, key point tracking module; 301, processor; 302, memory; 303, communication bus. Detailed Implementation

[0046] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0047] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0048] Firstly, please refer to Figure 1 and Figure 2 This application provides a method for labeling event data based on neuromorphic vision, used to label cellular events acquired using an event camera. The method includes the following steps:

[0049] S1. Obtain the event data stream to be processed, and divide the event data stream according to the pre-defined time window to generate the window event set to be processed;

[0050] S2. Perform noise filtering on each window event set and retain window events with a remaining number greater than a preset first threshold to form a valid event set.

[0051] S3. Based on unsupervised clustering and / or feature detection, extract key point information of each effective event set according to preset size parameters. The size parameters are set based on a pre-calibrated reference size.

[0052] S4. Track the trajectory of key point information and configure trajectory IDs to generate trajectory information to mark the event data stream.

[0053] Specifically, the event data labeling method based on neuromorphic vision refers to a technical solution that utilizes event data acquired by an event camera and processes it directly in the event domain to identify and track cellular events. The event data stream refers to the continuous data sequence output in real time by the event camera, containing pixel coordinates, timestamps, and polarity information. A pre-defined time window is used to divide the continuous event data stream into discrete window event sets; its size affects the granularity and efficiency of data processing. Noise filtering refers to the process of filtering events in the window event set to remove invalid or interfering events that do not represent the target cellular event. Various filtering techniques can be used, such as filtering based on the spatiotemporal distribution of events or filtering based on event polarity, primarily to improve the purity of the data in subsequent processing. Unsupervised clustering and / or feature detection refers to the technical means used to identify local event sets or specific patterns representing cellular events from the valid event set. Various methods can be used, such as cluster analysis based on the spatiotemporal distribution of events or detection based on local event patterns, primarily to locate and identify cellular events. The preset size parameters refer to the set of values ​​used to limit or constrain the range or shape of keypoint recognition during unsupervised clustering or feature detection. They are set based on a pre-calibrated baseline size, primarily to ensure accurate recognition according to the actual size of cell events. Keypoint information refers to the abstract description of the identified cell events after unsupervised clustering and / or feature detection. It can include location information, size information, time information, and event activity information, mainly to provide a structured description of cell events. Trajectory tracking refers to the process of associating keypoint information extracted from different time windows to reconstruct the movement path of cell events. It can employ various tracking algorithms, such as methods based on neighboring keypoint association or methods based on motion state estimation, primarily to obtain a continuous set of cell events. In this embodiment, the time window and baseline size are hyperparameters calibrated by analyzing test data before processing the event data stream.

[0054] More specifically, the first quantity threshold can be considered as the expected number of cell microspheres (corresponding to cell events) appearing in the image, which can be set according to the density of cell microspheres. In the embodiments of this application, 10 are preferred.

[0055] More specifically, the working principle of this application is as follows: First, the raw event data stream acquired by the event camera contains high temporal resolution event information. For ease of processing, the continuous event data stream is divided into a series of discrete window event sets based on pre-calibrated time windows. Next, noise filtering is performed on each window event set to remove invalid events, and windows containing a sufficient number of valid events are selected to form valid event sets. Then, for these valid event sets, unsupervised clustering and / or feature detection techniques are used, combined with size parameters set based on a pre-calibrated baseline size, to extract key point information representing cell events. Finally, the key point information extracted from different time windows is correlated, and a trajectory tracking algorithm connects key points belonging to the same cell event to form a complete motion trajectory. Each trajectory is assigned a unique identifier, thus completing the labeling of cell events. The entire process is performed directly in the event domain, fully utilizing the high temporal accuracy of the event camera, and improving the accuracy of labeling through noise filtering and key point extraction based on size parameters. Trajectory tracking provides dynamic information about cell events.

[0056] This application's method fully leverages the high temporal accuracy of event cameras for cell labeling, avoiding the temporal resolution loss inherent in traditional image-assisted labeling methods. Through noise filtering, it effectively removes noise from event data, improving the purity of the labeling results. Then, based on unsupervised clustering and / or feature detection, as well as keypoint extraction of size parameters, it can more accurately identify and locate cell events, distinguishing between target events and noisy events. The trajectory tracking process provides continuous motion information of cell events. The entire method achieves high-precision and high-efficiency labeling of cell events, reducing manual intervention and improving data processing efficiency.

[0057] In some preferred embodiments, in step S2, the noise filtering process is performed based on one or more cascaded event noise filters, which include background noise filters and refractory period filters.

[0058] Specifically, the event noise filter is used to filter noise in event data. The noise filtering process can use a background noise filter, a refractory period filter, or other event noise filters, or a cascaded form of them. In the embodiments of this application, the noise filtering process is preferably based on a cascaded background noise filter and a refractory period filter.

[0059] More specifically, a background noise filter refers to a technique used to identify and remove background-related interference events from event data. It can be implemented using methods based on spatial local density analysis. For example, by calculating the number of events for each event within a certain spatial neighborhood and comparing it to a preset threshold, events below the threshold are considered background noise. A refractory period filter refers to a technique used to identify and remove spurious events or noise that may occur when an event camera sensor enters its refractory period after detecting an event. It can be implemented using methods based on temporal local density analysis. For example, by checking whether other events occur at the same pixel or neighboring pixels within a very short time window after an event occurs, if so, subsequent events may be refractory period noise and are filtered out. Cascading refers to connecting two or more filters in series, with the output of the previous filter serving as the input of the next filter, to achieve step-by-step filtering processing.

[0060] Through the above design, the method of this application can effectively remove noise from different sources in event data, including background noise and noise caused by the refractory period effect, improve the thoroughness of noise filtering, thereby improving the quality of the effective event set, providing a cleaner data foundation for subsequent key point extraction and trajectory tracking, and improving the accuracy and reliability of the entire event data labeling method.

[0061] More specifically, the background noise filter has a first filtering time window, and the refractory period filter has a second filtering time window. In the background noise filter, if no other events exist within the 3×3 neighborhood of an event within the first filtering time window, the event is considered noise and will be filtered out by the background noise filter. In the refractory period filter, if an event occurs repeatedly at a certain position within the second filtering time window Δ, the event is considered a sudden event and will be filtered out by the refractory period filter.

[0062] Preferably, both the first filtering time window and the second filtering time window are set based on the aforementioned time window; wherein, the first filtering time window is the same as the time window, and the second filtering time window is half of the time window.

[0063] In some preferred embodiments, in step S3, the size parameter includes distance to the boundary. The process of extracting key point information of each valid event set based on unsupervised clustering and according to preset size parameters includes:

[0064] S31. Cluster the events in the effective event set based on the unsupervised clustering algorithm to obtain clustering results containing several event clusters;

[0065] S32. Filter out event clusters in each clustering result whose number of events is less than a preset second threshold;

[0066] S33. Filter outlier events from the remaining event clusters based on distance boundaries;

[0067] S34. Output key information based on the event clusters after filtering outlier events.

[0068] Specifically, unsupervised clustering algorithms refer to algorithms that divide data into different groups by analyzing the inherent structure of the data without prior knowledge of the category labels. These algorithms can be implemented using DBSCAN, K-Means, Mean Shift, etc. An effective event set refers to a window of events that has undergone initial noise filtering and contains a sufficient number of events; it forms the basis for subsequent processing. An event cluster is a set formed by grouping spatially and temporally adjacent event points using a clustering algorithm. A preset second quantity threshold is a quantitative standard used to determine whether an event cluster is a noisy cluster; its setting can be based on empirical estimation or statistical analysis of the size of noisy event clusters. Distance boundaries are distance standards used to determine whether events within a cluster are outliers; their setting considers the baseline size S to accommodate different target sizes. In this embodiment, the second quantity threshold is preferably set to 20.

[0069] Specifically, the above steps work as follows: First, unsupervised clustering algorithms are used to cluster events in the effective event set, organizing discrete event points into structurally meaningful clusters, forming preliminary clustering results. These clusters may correspond to potential cell events or noise. Next, noise clusters in the clustering results are filtered out by using a preset second threshold to filter out event clusters with fewer events than the threshold. This is because noise events or events generated by non-target objects are usually few in number, forming smaller clusters. Filtering these small clusters can effectively remove most noise interference, retaining clusters that are more likely to be real cell events. Further, outlier filtering is performed on the remaining event clusters. Even the retained clusters may contain some outlier event points far from the cluster center. By comparing the distances of each event point within a cluster to the corresponding cluster center based on a preset distance boundary, event points with distances greater than the distance boundary are identified as outliers and filtered out, effectively removing abnormal points within the clusters, making the remaining event points more representative of the core area of ​​cell events. Finally, key point information is output based on the event clusters after noise cluster filtering and outlier event filtering. Since these clusters have removed most of the noise and outliers, the keypoint information calculated based on them will be more accurate and reliable, providing high-quality input data for subsequent trajectory tracking and cell event labeling. This process works in conjunction with the preceding noise filtering step to further purify the data, providing more reliable input for subsequent trajectory tracking.

[0070] Preferably, the distance to the boundary is set using the following formula:

[0071] B = (S / 10 + 1) * 10 (1)

[0072] Where B is the distance to the boundary and S is the reference dimension.

[0073] It should be noted that if the unsupervised clustering algorithm uses the Mean Shift algorithm, the hyperparameter bandwidth in the algorithm can be directly set to the value of the distance boundary mentioned above.

[0074] In some preferred embodiments, in step S3, the size parameters include the minimum circle radius, the maximum circle radius, and the minimum circle center distance. The process of extracting key point information for each valid event set based on feature detection and according to preset size parameters includes:

[0075] S31' Convert the events in each valid event set into a two-dimensional edge graph and obtain the corresponding event timestamps;

[0076] S32', Based on the minimum circle radius, maximum circle radius, and minimum circle center distance, the edge map is inspected using Hough circle detection to label cell events based on circles;

[0077] S33' Outputs key information based on circular cell events and event timestamps.

[0078] Specifically, step S31' preferably involves reconstructing the optical structure information of events in the spatiotemporal neighborhood to obtain an edge map. Reconstructing the optical structure information of events in the spatiotemporal neighborhood means converting discrete, asynchronous event data into a continuous or semi-continuous representation that reflects local brightness changes or edge features, considering the time and spatial location of the event and information about neighboring events. This can be achieved using techniques such as image frame generation based on event accumulation, surface fitting based on event flow, or feature descriptor calculation based on local event activity. Converting the event into a two-dimensional edge map involves further processing the reconstructed optical structure information to generate a two-dimensional image representation, where pixel values ​​reflect whether there are edges or areas with drastic brightness changes at the corresponding spatial location. This can be achieved by applying edge detection operators such as Sobel, Prewitt, or Canny to process the reconstructed optical structure information. Hough circle detection is a feature extraction technique for detecting specific shapes (here, circles) in an image. Its principle is to map points in the image space to a parameter space and determine the geometric structure conforming to a specific shape in the image space by finding cumulative peaks in the parameter space.

[0079] Specifically, the above processing first converts the event data into a two-dimensional edge map, allowing the application of mature image processing techniques. This conversion process considers the spatiotemporal neighborhood information of the event, transforming discrete events into edge representations with a certain degree of spatial continuity, laying the foundation for subsequent shape detection, while preserving the original event's timestamp information. After acquiring the edge map, the Hough circle detection algorithm actively searches for circular structures that conform to a preset size range (minimum circle radius, maximum circle radius) and spatial distribution (minimum circle center distance). These parameters are set based on pre-calibrated reference dimensions to constrain the detection results, ensuring they conform to the expected characteristics of cellular events. Hough circle detection directly utilizes the circular shape features that cellular events may exhibit, complementing spatial density-dependent clustering methods. The detected circular labels represent the spatial location and size of the cellular events. Finally, these detected circular labels are associated with the corresponding event timestamps, outputting keypoint information containing attributes such as spatial location, size, and time.

[0080] In this way, the method of this application can more accurately identify and locate cell events with circular features, especially suitable for scenarios where cell events appear as clear circular outlines in event data. This shape-based detection method, combined with or as an alternative to spatial density-based clustering methods, enhances the ability of keypoint extraction to identify targets with specific shapes, improves the accuracy and robustness of keypoint extraction, and thus provides more reliable input for subsequent trajectory tracking.

[0081] Preferably, the formula for setting the minimum circle radius is as follows:

[0082] R min =max(5*(S / 10-1),10) (2)

[0083] The formula for setting the maximum circle radius is as follows:

[0084] R max = 5*(S / 10+2) (3)

[0085] Dist min =R max *2 / 3 (4)

[0086] Among them, R min R is the minimum circle radius. max Dist is the radius of the largest circle. min This is the minimum center distance.

[0087] It should be noted that the edge map is actually a two-dimensional histogram of events. Its transformation process is based on the normalized weighted sum of the number of all triggered events in the effective event set corresponding to each pixel within the statistical time window and the maximum value of the corresponding image pixel.

[0088] In some preferred embodiments, a further step is included between step S31' and step S32':

[0089] S3A': Downsample the edge map.

[0090] Specifically, downsampling reduces the number of pixels in the edge map, thereby reducing the data volume and resolution. The downsampled edge map is then used for Hough circle detection. By performing Hough circle detection on a lower-resolution edge map, the required computational resources and time are significantly reduced, improving processing efficiency. Furthermore, the downsampling process typically has a smoothing effect, which helps suppress subtle noise and irregularities in the edge map, making Hough circle detection less sensitive to noise, improving robustness and accuracy, and allowing for more stable identification of the circular structure of cellular events.

[0091] In some preferred embodiments, key point information includes centroid coordinates, pixel size, timestamp, and event weight.

[0092] Specifically, the centroid coordinates refer to the center position of the cell event on the image plane, which can be represented by pixel coordinate pairs (x, y). The pixel size refers to the size occupied by the cell event on the image plane, which can be represented by equivalent diameter or bounding box size, etc. The timestamp refers to the time point in time when the cell event occurred, which can be represented by a microsecond or nanosecond time value output by the event camera. The event weight refers to the number of events associated with the keypoint or its activity level, which can be represented by the original event count constituting the keypoint or a value calculated based on the event activity intensity.

[0093] More specifically, the method of this application provides a richer and more accurate description of cell events by including data in four dimensions—centroid coordinates, pixel size, timestamp, and event weight—in the key point information.

[0094] More specifically, centroid coordinates provide the spatial basis for cell events and serve as the starting point for trajectory tracking. Pixel size supplements the size characteristics of cell events, helping to distinguish targets of different sizes or identify size variations of the same target, which is crucial for handling cell deformation or differentiating similar targets. Timestamps leverage the high temporal resolution of the event camera to provide a basis for accurate time correlation and velocity calculation, particularly suitable for trajectory reconstruction in high-speed flowing scenes. Event weights provide information on the reliability or importance of key points, which can be used for weighted processing or as a confidence reference during trajectory tracking, enhancing the ability to handle noisy or incomplete data.

[0095] It should be noted that in the processing method of obtaining key point information based on unsupervised clustering, the centroid coordinates are the centroid coordinates of the event cluster, which is the average coordinate of the events within the cluster. The pixel size can be represented by the length and width or the circle size (radius or diameter). The timestamp can be represented by the latest timestamp of the events within the cluster, and the event weight is the amount of event data within the cluster.

[0096] It should be noted that in the processing method of obtaining key point information based on feature detection, the centroid coordinates are the coordinates of the center of the detected circle, the pixel size is the circle size (radius or diameter), the timestamp can be the event timestamp extracted earlier, and the weight w is the number of votes for the circle center.

[0097] It should be noted that unsupervised clustering and feature detection can be used individually or in parallel, and using them in parallel can play a role in information complementarity.

[0098] In some preferred embodiments, step S4 includes:

[0099] S41. Group the key point information according to the pixel distance and time difference between key point information, and assign trajectory IDs to obtain preliminary trajectory information;

[0100] S42. For each preliminary trajectory information, merge the key point information whose time difference is less than the preset first time threshold to obtain trajectory information to mark the event data stream.

[0101] Specifically, pixel distance refers to the spatial distance between two key points on the image plane. Temporal difference refers to the difference between the timestamps corresponding to two key points.

[0102] Specifically, step S41 obtains preliminary trajectory information by grouping keypoint information based on pixel distance and time difference and assigning trajectory IDs. This step utilizes the spatial and temporal correlation of keypoints in event camera data. Cell events typically appear as spatially close keypoints on the image plane under consecutive timestamps. By simultaneously considering pixel distance and time difference, keypoints belonging to the same potential trajectory can be effectively grouped together initially, and unique trajectory IDs can be assigned to these preliminary trajectory segments, thus laying the foundation for subsequent trajectory refinement. Next, step S42 merges keypoint information with a time difference less than a preset first time threshold for each preliminary trajectory information to obtain the final trajectory information. Step S41 may generate multiple short trajectory segments belonging to the same cell due to the intermittency of keypoint detection or the complexity of cell motion. Step S42 merges preliminary trajectory segments that are temporally continuous (time difference less than the threshold) by setting a time threshold. This merging process connects previously scattered trajectory segments to form a more complete and longer trajectory, thereby improving the accuracy and continuity of trajectory tracking and ultimately generating precise trajectory information for event data labeling.

[0103] In some implementations, step S41 can be implemented in the following manner:

[0104] Given input keypoint information, if the historical trajectory is empty, a new trajectory is created and assigned a trajectory ID, with the input keypoint as the starting point of the trajectory. If the historical trajectory is not empty, the input keypoint information is matched against each historical trajectory. If a match is successful, the keypoint information is associated with the corresponding trajectory to gradually form preliminary trajectory information. The matching process between keypoint information and historical trajectories is as follows.

[0105] The process iterates through historical trajectories, extracting the tail keypoints from each trajectory. It calculates the pixel distance and time difference between the keypoint information and each tail keypoint. If the time difference is less than a preset second time threshold, and the pixel distance is less than the largest size among all keypoints in the corresponding historical trajectory, then that historical trajectory is considered a candidate trajectory. Finally, the candidate trajectory with the smallest pixel distance is taken as the matching result for that keypoint information, and this keypoint information is then incorporated into the historical trajectory. If no candidate trajectory exists during the matching process, a new trajectory is created, a trajectory ID is assigned to the keypoint information, thus forming a new historical trajectory, and this keypoint serves as the starting point of the new historical trajectory.

[0106] Therefore, the above processing method gradually incorporates the information of each key point into different trajectories. After the matching of all key point information is completed, these allocated historical trajectories are determined as preliminary trajectory information.

[0107] In some implementations, in step S42, merging the key point information is based on the following formula:

[0108] (5)

[0109] (6)

[0110] (7)

[0111] (8)

[0112] Where c1 and c2 are the centroid coordinates of the two keypoints to be merged, w1 and w2 are the event weights of the two keypoints to be merged, t1 and t2 are the timestamps of the two keypoints to be merged, and s1 and s2 are the pixel sizes of the two keypoints to be merged. new w new t new and s new These are the centroid coordinates, event weights, timestamps, and pixel dimensions of the merged keypoint information.

[0113] In some preferred embodiments, the first time threshold and the second time threshold are preferably set based on a time window, respectively satisfying:

[0114] T1 = α·△T, 0 < α < 1 (9)

[0115] T2 = β·△T, β > 1 (10)

[0116] Where T1 is the first time threshold, T2 is the second time threshold, ΔT is the time window, and α and β are preset time coefficients that can be set according to usage requirements, such as α set to 0.5 and β set to 15.

[0117] In some preferred embodiments, the calibration process for the time window and the reference size includes:

[0118] A1. Acquire the event data stream for testing, and mark the event data stream for testing based on the preset initial time window and initial reference size to obtain pre-detection trajectory information;

[0119] A2. Calculate the average velocity information of cell events based on the pre-detection trajectory information;

[0120] A3. Calculate the calibration time window based on the ratio of the expected pixel movement speed to the average speed information of the set cell events;

[0121] A4. Calculate the average size information corresponding to cell events based on the pre-detection trajectory information, and use it as the baseline size.

[0122] Specifically, the test event data stream refers to a segment of event data used for parameter calibration. This can be a segment extracted from a complete event data stream or data specifically collected for testing. The preset initial time window refers to a pre-set time window value used during initial labeling; this can be an empirical value or a randomly set value. The preset initial reference size refers to a pre-set reference size value used during initial labeling; this can be an empirical value or a randomly set value. Pre-detection trajectory information refers to trajectory data obtained through the initial labeling process, containing information such as the movement trajectory and size of cell events identified under the initial parameters. The average velocity information of cell events refers to the average moving velocity of the cell on the pixel plane, statistically calculated based on the pre-detection trajectory information. The set expected pixel moving speed of cell events refers to a target velocity value set by the user or system according to actual application requirements, representing the desired pixel distance the cell will move within each time window. The average size information corresponding to cell events refers to the average size of the cell on the pixel plane, statistically calculated based on the pre-detection trajectory information.

[0123] More specifically, step A1 acquires a test event data stream and performs preliminary labeling on this data using a preset initial time window and initial reference size. The acquired pre-detection trajectory information includes the movement trajectory and key point information of cell events under the initial parameters, providing a data source for subsequent parameter adjustments. Step A2 estimates the average moving speed of the cell on the pixel plane by analyzing the changes of key points in the pre-detection trajectory over time. This average moving speed reflects the actual movement characteristics of the cell in the event camera's field of view under the current initial parameters and is a key basis for calibrating the time window. Step A3 calculates and calibrates the final time window based on the ratio of the set expected pixel movement speed of the cell event to the average speed information calculated in step A2. By comparing the actually measured average speed with the expected speed and adjusting the size of the time window based on their proportional relationship, the calibrated time window can better adapt to the actual movement speed of the cell, ensuring that an appropriate amount of cell event information can be captured within a time window, thereby optimizing the segmentation effect of the event data stream. Step A4 calculates the average size information corresponding to cell events based on the pre-detection trajectory information obtained in step A1, and uses this average size information as the final reference size. Keypoints in the pre-detection trajectory information may contain cell event size information. By statistically averaging these size information, the average pixel size of the cells can be obtained. Using this average size as the reference size provides a data-driven basis for setting size parameters in the subsequent keypoint extraction process, enabling the keypoint extraction process to more accurately identify cell events with this average size feature and improve the accuracy of keypoint extraction.

[0124] Through the above scheme, the method of this application realizes adaptive calibration of time window and reference size, improves the automation level of event data labeling method and adaptability to different cell types and motion states. By using the preliminary labeling results to adjust key parameters, the parameter settings can more accurately reflect the actual cell motion speed and size characteristics, thereby improving labeling accuracy and robustness, and better leveraging the high temporal resolution advantage of event cameras.

[0125] In some preferred embodiments, the pre-detection trajectory information is based on multiple pre-detection key point information;

[0126] Step A2 includes:

[0127] A21. Extract pre-detection trajectory information where the number of pre-detection key point information is greater than a preset third quantity threshold, and calculate the average speed information of the pre-detection key point information based on these pre-detection trajectory information.

[0128] Step A4 includes:

[0129] A41. Extract pre-detection trajectory information where the number of pre-detection key point information is greater than the third quantity threshold, and calculate the average size information of the pre-detection key point information based on these pre-detection trajectory information as the reference size.

[0130] Specifically, the third quantity threshold refers to the minimum number of keypoints required to filter pre-detection trajectory information. This threshold is set to distinguish reliable trajectories containing sufficient information to represent the true motion and size characteristics of cellular events from unreliable trajectories with too few keypoints due to noise, occlusion, or transient occurrences. This threshold can be set according to the actual application scenario, the characteristics of the event camera, and the expected noise level; in this embodiment, the third quantity threshold can be set to 5.

[0131] Specifically, the above implementation introduces a screening step: when calculating the average velocity information of cell events, instead of directly using all acquired pre-detection trajectory information, the number of pre-detection keypoints contained in each pre-detection trajectory is first checked. Only when the number of keypoints contained in a pre-detection trajectory is greater than a preset third threshold is the trajectory retained for calculation. Based on these screened pre-detection trajectory information with a sufficient number of keypoints, the average velocity information of the pre-detection keypoints is calculated. Similarly, the same screening strategy is used when calculating the average size information corresponding to cell events and using it as the baseline size. Pre-detection trajectory information with a number of pre-detection keypoints greater than the preset third threshold is extracted, and the average size information of the pre-detection keypoints is calculated based on these screened trajectory information. In this way, the interference of unreliable trajectories containing a small number of keypoints due to incomplete data or noise interference on the calculation of average velocity and average size is eliminated. This screening process ensures that the average velocity and average size data used for calibrating the time window and baseline size are more accurate and reliable.

[0132] Through the above processing, the method of this application avoids the interference of unreliable trajectories containing a few key points on the calculation of average velocity and average size. Therefore, the calculated average velocity and average size can more accurately reflect the true characteristics of cellular events, making the calibration of time windows and baseline sizes more precise and improving the accuracy of subsequent event data labeling.

[0133] In some preferred embodiments, in step A2, the average speed information can be calculated by extracting key point information with a certain number of intervals from the pre-detection trajectory information. For example, the average speed information can be calculated by extracting key point information from each pre-detection trajectory information with 3 key points as the interval and extracting the corresponding coordinates and timestamps.

[0134] In some preferred embodiments, the formula for calculating the time window in step A3 is:

[0135] (11)

[0136] Where D is the expected pixel movement speed of the cell event. This is average speed information.

[0137] In some preferred embodiments, in step A4, for the circle size, the average size information can be the diameter, and for the rectangle size, the average size information can be the larger of the length and width dimensions.

[0138] In some preferred embodiments, step A1 involves using the event data stream for testing as the event data stream to be processed, the initial time window as the time window, and the initial reference size as the reference size, and using the trajectory information obtained from steps S1 to S4 as the pre-detection trajectory information.

[0139] Specifically, steps S1-S4 refer to applying a specific event data labeling method, which includes steps such as segmenting the event data stream, noise filtering, extracting key point information, and trajectory tracking and configuring trajectory IDs. Pre-detection trajectory information refers to trajectory data obtained by executing the aforementioned event data labeling method, based on the test event data stream and preset initial parameters (initial time window and initial reference size). This data is used for subsequent calibration calculations.

[0140] Specifically, the above design works by using the event data labeling method itself, which will be used in practical applications, to generate pre-calibration trajectory information. First, the event data stream used for testing and calibration is taken as input to the labeling method. Then, during each step of the labeling method, a preset initial time window and a preset initial reference size are used as the reference size parameter. Next, all steps of the labeling method are executed completely, including segmenting the event data stream based on the time window, filtering noise from the segmented event set, extracting key point information based on the reference size, and tracking and assigning trajectory IDs to the key point information, ultimately outputting the trajectory information. Finally, the trajectory information obtained in this way is designated as the pre-calibration trajectory information. By using the same labeling process as in practical applications to generate the pre-calibration trajectory information, it is ensured that the data used for calibration accurately reflects the behavior and characteristics of the labeling method when processing event data. Based on this highly consistent pre-detection trajectory information, subsequent calculations of average velocity and average size can yield more accurate statistical results. This makes the final calibrated time window and baseline size parameters more suitable for optimizing the performance of the event data labeling method in practical applications, improving the reliability and effectiveness of the calibration results. This approach tightly couples the data generation process used for calibration with the method to be calibrated, enabling the calibration results to more accurately reflect the method's performance in real-world scenarios, thereby improving the applicability and accuracy of the calibration parameters.

[0141] In the above embodiments, step A1 can be performed in the following manner:

[0142] B1. Obtain the event data stream for testing, and divide the event data stream for testing based on the preset initial time window to generate a set of window events for testing;

[0143] B2. Perform noise filtering on each test window event set and retain the test window event sets with a remaining number of events greater than the preset first number threshold to form a valid pre-detection event set.

[0144] B3. Based on unsupervised clustering, extract key point information of each effective pre-detection event set according to the reference size parameters set based on the initial baseline size;

[0145] B4. Track the key point information of the effective pre-inspection event set and configure the trajectory ID to generate pre-inspection trajectory information.

[0146] In some implementations, the reference size parameter used in step B3 can also be replaced by a setting based on the screen size, such as setting the target object (the cell event to be tested) to occupy 1 / 8 of the shortest side of the screen.

[0147] In some implementations, after generating trajectory information in step S4, the method of this application can archive key point information by event tagging. For example, based on the centroid coordinates of the key point information, the size of the key point is appropriately enlarged to form a region of interest (ROI), and all events covered by the ROI are marked with the key point label labelID. Alternatively, all events that participate in forming the key point information are marked with the key point label labelID, where the value in the labelID dimension is the trajectory ID, and different trajectory IDs correspond to different target objects (cell microspheres).

[0148] Secondly, please refer to Figure 3 Some embodiments of this application also provide an event data labeling system based on neuromorphic vision for labeling cellular events acquired using an event camera. The system includes:

[0149] The preprocessing module 201 is used to acquire the event data stream to be processed, and to divide the event data stream according to a pre-calibrated time window to generate a set of window events to be processed. It is also used to perform noise filtering on each set of window events and to retain window events with a remaining number of events greater than a preset first number threshold to form a valid event set.

[0150] The key point detection module 202 is used to extract key point information of each effective event set based on unsupervised clustering and / or feature detection, according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size;

[0151] The key point tracking module 203 is used to track the key point information, configure the trajectory ID, and generate trajectory information to mark the event data stream.

[0152] This system fully leverages the high temporal accuracy of event cameras for cell labeling, avoiding the temporal resolution loss inherent in traditional image-assisted labeling methods. Through noise filtering, it effectively removes noise from event data, improving the purity of the labeling results. Then, based on unsupervised clustering and / or feature detection, as well as keypoint extraction of size parameters, it can more accurately identify and locate cell events, distinguishing between target events and noisy events. The trajectory tracking process provides continuous motion information of cell events. The entire system achieves high-precision and high-efficiency labeling of cell events, reducing manual intervention and improving data processing efficiency.

[0153] In some implementations, the system also includes a tagging module for archiving key point information by means of event tagging.

[0154] Thirdly, please refer to Figure 4 Some embodiments of this application also provide a schematic diagram of the structure of an electronic device. This application provides an electronic device including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanism (not shown). The memory 302 stores computer-readable instructions that can be executed by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.

[0155] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program performs the method in any optional implementation of the above embodiments. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0156] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0157] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0158] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0159] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0160] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for labeling event data based on neuromorphic vision, used to label cellular events acquired using an event camera, characterized in that, The method includes the following steps: S1. Obtain the event data stream to be processed, and divide the event data stream according to a pre-defined time window to generate a set of window events to be processed; S2. Perform noise filtering on each window event set and retain window events with a remaining number greater than a preset first threshold to form a valid event set. S3. Based on unsupervised clustering and / or feature detection, extract key point information of each effective event set according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size; S4. Track the key point information and configure the trajectory ID to generate trajectory information to mark the event data stream; The calibration process for the time window and the reference size includes: A1. Acquire the event data stream for testing, and mark the event data stream for testing based on a preset initial time window and initial reference size to obtain pre-detection trajectory information; A2. Calculate the average velocity information of cell events based on the pre-detection trajectory information; A3. Calculate and calibrate the time window based on the ratio of the expected pixel movement speed of the set cell event to the average speed information; A4. Calculate the average size information corresponding to the cell events based on the pre-detection trajectory information, and use it as the reference size.

2. The event data tagging method based on neuromorphic vision according to claim 1, characterized in that, In step S2, the noise filtering process is performed based on one or more cascaded event noise filters, which include background noise filters and refractory period filters.

3. The event data tagging method based on neuromorphic vision according to claim 1, characterized in that, In step S3, the size parameters include distance to the boundary. The process of extracting key point information for each valid event set based on unsupervised clustering and according to preset size parameters includes: S31. Cluster the events in the effective event set based on the unsupervised clustering algorithm to obtain clustering results containing several event clusters; S32. Filter out event clusters in each clustering result whose number of events is less than a preset second threshold; S33. Filter outlier events from the remaining event clusters based on the distance boundary; S34. Output key information based on the event clusters after filtering outlier events.

4. The event data labeling method based on neuromorphic vision according to claim 1, characterized in that, In step S3, the size parameters include the minimum circle radius, the maximum circle radius, and the minimum circle center distance. The process of extracting key point information for each valid event set based on feature detection and according to the preset size parameters includes: S31' Convert the events in each valid event set into a two-dimensional edge graph and obtain the corresponding event timestamps; S32', Based on the minimum circle radius, maximum circle radius, and minimum circle center distance, examine the edge map using Hough circle detection to label cell events based on circles; S33' Outputs the key point information based on circular cell events and the event timestamps.

5. The event data tagging method based on neuromorphic vision according to claim 4, characterized in that, The step between step S31' and step S32' includes: S3A', Perform downsampling processing on the edge map.

6. The event data tagging method based on neuromorphic vision according to claim 1, characterized in that, Step S4 includes: S41. Group the key point information according to the pixel distance and time difference between key point information, and assign trajectory IDs to obtain preliminary trajectory information; S42. For each preliminary trajectory information, merge the key point information whose time difference is less than a preset first time threshold to obtain the trajectory information to mark the event data stream.

7. An event data labeling system based on neuromorphic vision, used to label cellular events acquired using an event camera, characterized in that, The system includes: The preprocessing module is used to acquire the event data stream to be processed, and to divide the event data stream according to a pre-calibrated time window to generate a set of window events to be processed. It is also used to perform noise filtering on each set of window events and to retain window events with a remaining number of events greater than a preset first number threshold to form a valid event set. The key point detection module is used to extract key point information of each effective event set based on unsupervised clustering and / or feature detection, according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size; The key point tracking module is used to track the trajectory of the key point information, configure the trajectory ID, and generate trajectory information to mark the event data stream; The calibration process for the time window and the reference size includes: A1. Acquire the event data stream for testing, and mark the event data stream for testing based on a preset initial time window and initial reference size to obtain pre-detection trajectory information; A2. Calculate the average velocity information of cell events based on the pre-detection trajectory information; A3. Calculate and calibrate the time window based on the ratio of the expected pixel movement speed of the set cell event to the average speed information; A4. Calculate the average size information corresponding to the cell events based on the pre-detection trajectory information, and use it as the reference size.

8. An electronic device, characterized in that, It includes a processor and a memory, the memory storing computer-readable instructions that, when executed by the processor, perform the steps of the method as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it performs the steps of the method as described in any one of claims 1-6.

Images