Event data marking method based on neuromorphic vision and related equipment

Through the event data labeling method of neuromorphic vision, using technical means such as noise filtering and unsupervised clustering, the problems of temporal resolution loss and inaccurate labeling in existing technologies are solved, and efficient and accurate cell event labeling is achieved.

CN120765963AActive Publication Date: 2025-10-10JIHUA LAB
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202511283236.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2025-09-09
Publication Date
2025-10-10
Estimated Expiration
2045-09-09

AI Technical Summary

Technical Problem

Existing methods rely on image labeling assistance in the process of event data labeling, which leads to reduced temporal resolution, inaccurate and susceptible labels, difficulty in distinguishing target object events from noise events, and cumbersome and costly labeling processes.

Method used

An event data labeling method based on neuromorphic vision is adopted to extract key point information through noise filtering, unsupervised clustering and feature detection, and automatic labeling is achieved by combining trajectory tracking, making full use of the high temporal precision characteristics of event cameras.

Benefits of technology

The purity and accuracy of labeling results are improved, manual intervention is reduced, and high-precision and high-efficiency cell event labeling is achieved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120765963A_ABST
    Figure CN120765963A_ABST
Patent Text Reader

Abstract

The invention relates to the technical field of flow cytometry sorting, and particularly discloses an event data marking method and related equipment based on neuromorphic vision, and the method comprises the following steps: obtaining a to-be-processed event data stream, segmenting the event data stream based on a pre-calibrated time window, and generating a to-be-processed window event set; performing noise filtering processing on each window event set, and reserving the window events with the residual event quantity greater than a preset first quantity threshold to form an effective event set; based on unsupervised clustering and / or feature detection, key point information of each effective event set is extracted according to preset size parameters; performing trajectory tracking on the key point information, configuring a trajectory ID, and generating trajectory information to mark the event data stream; according to the method, the high time precision characteristic of the event camera can be fully utilized to realize cell marking, and the time resolution loss caused by a traditional image-based auxiliary marking method is avoided.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of flow cell sorting technology, and in particular to an event data marking method based on neuromorphic vision and related equipment. Background Art

[0002] In the field of imaging flow cytometry, neuromorphic visual event cameras, with their high temporal resolution and low data rate, are gradually replacing traditional machine vision cameras. They provide a new technical path for rapid sensing and real-time sorting of cells in high-speed flows. The application of event data shows great promise, especially in creating efficient datasets, laying the foundation for the subsequent training of sorting and classification models.

[0003] However, in the process of event data labeling, existing methods mainly rely on image-based labeling-assisted methods, such as converting event data into a two-dimensional grid to generate an image representation and then manually labeling it, or using high-speed camera images to assist in labeling. This labeling method significantly reduces the temporal resolution of events and fails to fully utilize the high temporal precision characteristics of event cameras, resulting in the inability of labels to accurately match the microsecond-level changes of events. At the same time, this method has difficulty in finely distinguishing between target object events and noise events, and the labeling results are inaccurate and susceptible to interference. The labeling process is cumbersome, requires a lot of manual intervention, is inefficient, and is costly.

[0004] There is currently no effective technical solution to the above problems. Summary of the Invention

[0005] The purpose of this application is to provide an event data labeling method and related equipment based on neuromorphic vision to realize automatic labeling of event data, so as to fully utilize the high time precision characteristics of event cameras, reduce labor costs and improve labeling efficiency.

[0006] In a first aspect, the present application provides an event data marking method based on neuromorphic vision, for marking cell events collected using an event camera, the method comprising the following steps: S1. Obtain an event data stream to be processed, and segment the event data stream based on a pre-calibrated time window to generate a window event set to be processed; S2. Perform noise filtering on each window event set, and retain window events whose number of remaining events is greater than a preset first number threshold to form a valid event set; S3. Extracting key point information of each valid event set based on unsupervised clustering and / or feature detection according to preset size parameters, where the size parameters are set based on a pre-calibrated reference size; S4. Track the key point information and configure a trajectory ID to generate trajectory information to mark the event data stream.

[0007] The method of the present application can make full use of the high temporal precision characteristics of the event camera to realize cell labeling, avoiding the loss of temporal resolution caused by traditional image-assisted labeling methods. It effectively removes noise from the event data through noise filtering, improves the purity of the labeling results, and then based on unsupervised clustering and / or feature detection and key point extraction of size parameters, it can more accurately identify and locate cell events and distinguish target events from noise events. The trajectory tracking process provides continuous motion information of cell events. The entire method achieves high-precision and high-efficiency labeling of cell events, reduces manual intervention, and improves data processing efficiency.

[0008] In the event data marking method based on neuromorphic vision, in step S2, noise filtering processing is performed based on a single or multiple cascaded event noise filters, and the event noise filters include a background noise filter and a refractory period filter.

[0009] In this example, the applied method can effectively remove noise from different sources in the event data, including background noise and noise caused by the refractory period effect, thereby improving the thoroughness of noise filtering and thus improving the quality of the effective event set, providing a purer data foundation for subsequent key point extraction and trajectory tracking, and improving the accuracy and reliability of the entire event data labeling method.

[0010] In the event data labeling method based on neuromorphic vision, in step S3, the size parameter includes a distance boundary, and the process of extracting key point information of each valid event set based on the preset size parameter based on unsupervised clustering includes: S31. Clustering the events in the valid event set based on an unsupervised clustering algorithm to obtain a clustering result containing several event clusters; S32, filtering event clusters whose event quantity in each clustering result is less than a preset second quantity threshold; S33, filtering outlier events from the remaining event clusters based on the distance boundary; S34. Output key point information based on the event cluster filtered out of the outlier events.

[0011] In the event data labeling method based on neuromorphic vision, in step S3, the size parameters include the minimum circle radius, the maximum circle radius, and the minimum circle center distance. Based on feature detection, the process of extracting key point information of each valid event set according to the preset size parameters includes: S31′, converting the events in each valid event set into a two-dimensional edge graph, and obtaining the corresponding event timestamps; S32', checking the edge map based on the Hough circle detection according to the minimum circle radius, the maximum circle radius and the minimum circle center distance, to label a cell event based on a circle; S33', outputting the key point information based on the circle cell event and the event timestamp.

[0012] The event data labeling method based on neuromorphic vision, wherein between the step S31' and the step S32', the method further comprises a step of: S3A', downsampling the edge map.

[0013] The event data labeling method based on neuromorphic vision, wherein the step S4 comprises: S41, grouping the key point information according to the pixel distance and the time difference between the key point information, and assigning a track ID to obtain preliminary track information; S42, for each preliminary track information, merging the key point information with a time difference less than a preset first time threshold, to obtain the track information to label the event data stream.

[0014] The event data labeling method based on neuromorphic vision, wherein the calibration process of the time window and the reference size comprises: A1, obtaining a test event data stream, and labeling the test event data stream based on a preset initial time window and an initial reference size, to obtain preliminary track information; A2, calculating average speed information of a cell event according to the preliminary track information; A3, calculating the time window according to a ratio of a preset expected pixel movement speed of a cell event to the average speed information; A4, calculating average size information corresponding to a cell event according to the preliminary track information, as the reference size.

[0015] In a second aspect, the present application further provides an event data labeling system based on neuromorphic vision, for labeling a cell event collected by an event camera, the system comprising: a preprocessing module, configured to obtain an event data stream to be processed, and split the event data stream based on a pre-calibrated time window to generate a window event set to be processed, and further configured to perform noise filtering processing on each window event set, and retain a window event with a remaining event number greater than a preset first number threshold to form an effective event set; a key point detection module, configured to extract key point information of each effective event set based on unsupervised clustering and / or feature detection according to a preset size parameter, the size parameter being set based on a pre-calibrated reference size; The key point tracking module is used to track the key point information and configure a track ID, and generate track information to mark the event data stream.

[0016] The system of the present application can fully utilize the high temporal precision characteristics of the event camera to realize cell labeling, avoiding the temporal resolution loss caused by the traditional image-assisted labeling method. It effectively removes the noise in the event data through noise filtering, improves the purity of the labeling results, and then based on unsupervised clustering and / or feature detection and key point extraction of size parameters, it can more accurately identify and locate cell events and distinguish target events from noise events. The trajectory tracking process provides continuous motion information of cell events. The entire system achieves high-precision and high-efficiency labeling of cell events, reduces manual intervention, and improves data processing efficiency.

[0017] In a third aspect, the present application further provides an electronic device comprising a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps of the method provided in the first aspect are executed.

[0018] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, runs the steps of the method provided in the first aspect above.

[0019] As can be seen from the above, the present application provides an event data labeling method and related equipment based on neuromorphic vision, wherein the method of the present application can make full use of the high temporal precision characteristics of the event camera to achieve cell labeling, avoiding the loss of temporal resolution caused by traditional image-assisted labeling methods. It effectively removes noise from event data through noise filtering, improves the purity of the labeling results, and then based on unsupervised clustering and / or feature detection and key point extraction of size parameters, can more accurately identify and locate cell events, and distinguish target events from noise events. The trajectory tracking process provides continuous motion information of cell events. The entire method achieves high-precision and high-efficiency labeling of cell events, reduces manual intervention, and improves data processing efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a flowchart of the event data labeling method based on neuromorphic vision provided in an embodiment of the present application.

[0021] Figure 2 A detailed flowchart of the event data labeling method based on neuromorphic vision provided in an embodiment of the present application.

[0022] Figure 3A schematic diagram of the structure of an event data marking system based on neuromorphic vision provided in an embodiment of the present application.

[0023] Figure 4 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0024] Reference numerals: 201, pre-processing module; 202, key point detection module; 203, key point tracking module; 301, processor; 302, memory; 303, communication bus. DETAILED DESCRIPTION

[0025] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present application provided in the drawings is not intended to limit the scope of the application for protection, but merely represents the selected embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without making creative work fall within the scope of protection of the present application.

[0026] It should be noted that similar reference numerals and letters represent similar items in the following drawings. Therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings. At the same time, in the description of this application, the terms "first", "second", etc. are only used to distinguish the description and should not be understood as indicating or implying relative importance.

[0027] First, please refer to Figure 1 and Figure 2 Some embodiments of the present application provide an event data marking method based on neuromorphic vision, which is used to mark cell events collected by an event camera. The method includes the following steps: S1. Obtain an event data stream to be processed, and split the event data stream based on a pre-calibrated time window to generate a window event set to be processed; S2. Perform noise filtering on each window event set, and retain window events whose number of remaining events is greater than a preset first number threshold to form a valid event set; S3. Based on unsupervised clustering and / or feature detection, extract key point information of each valid event set according to preset size parameters, where the size parameters are set based on a pre-calibrated reference size; S4. Track the key point information and configure the trajectory ID to generate trajectory information to mark the event data stream.

[0028] Specifically, the event data labeling method based on neuromorphic vision refers to a technical solution that uses event data collected by an event camera to be processed directly in the event domain to identify and track cell events. The event data stream refers to a continuous data sequence output by the event camera in real time, which contains pixel coordinates, timestamps and polarity information. The pre-calibrated time window is used to divide the continuous event data stream into discrete window event sets, and its size affects the granularity and efficiency of data processing. Noise filtering refers to the process of screening events in the window event set to remove invalid or interfering events that do not represent target cell events. It can use a variety of filtering techniques, such as filtering based on the spatiotemporal distribution of events or filtering based on event polarity, mainly to improve the data purity of subsequent processing. Unsupervised clustering and / or feature detection refers to a technical means for identifying local event sets or specific patterns representing cell events from the valid event set. It can use a variety of methods, such as cluster analysis based on the spatiotemporal distribution of events or detection based on local event patterns, mainly to locate and identify cell events. The preset size parameter refers to a numerical set used to limit or constrain the key point recognition range or shape during unsupervised clustering or feature detection. The setting basis is a pre-calibrated reference size, mainly for accurately identifying according to the actual size of the cell event. Key point information refers to an abstract description of the cell event identified after unsupervised clustering and / or feature detection, which can include position information, size information, time information and event activity information, mainly for providing a structured description of the cell event. Trajectory tracking refers to the process of associating the key point information extracted in different time windows to reconstruct the motion path of the cell event. It can adopt a variety of tracking algorithms, such as methods based on the association of adjacent key points or methods based on motion state estimation, mainly for obtaining a continuous cell event set of cell events. In an embodiment of the present application, the time window and reference size are hyperparameters that are calibrated by analyzing the test data before processing the event data stream.

[0029] More specifically, the first quantity threshold can be considered as the expected number of cell microspheres (corresponding to cell events) appearing in the picture, which can be set according to the density of the cell microspheres. In the embodiment of the present application, 10 is preferred.

[0030] More specifically, the method of the present application works by first acquiring the raw event data stream captured by the event camera, which contains event information with high temporal resolution. For ease of processing, the continuous event data stream is divided into a series of discrete window event sets based on pre-calibrated time windows. Next, each window event set is subjected to noise filtering to remove invalid events, and windows containing a sufficient number of valid events are screened to form a valid event set. Then, for these valid event sets, unsupervised clustering and / or feature detection techniques are used, combined with size parameters set based on pre-calibrated reference sizes, to extract key point information representing cell events. Finally, the key point information extracted from different time windows is associated, and the key points belonging to the same cell event are connected using a trajectory tracking algorithm to form a complete motion trajectory. A unique identifier is assigned to each trajectory, thereby completing the labeling of cell events. The entire process is carried out directly in the event domain, fully utilizing the high temporal precision of the event camera, and improving the accuracy of labeling through noise filtering and key point extraction based on size parameters. Trajectory tracking provides dynamic information of cell events.

[0031] The method of the present application can make full use of the high temporal precision characteristics of the event camera to realize cell labeling, avoiding the loss of temporal resolution caused by traditional image-assisted labeling methods. It effectively removes noise from the event data through noise filtering, improves the purity of the labeling results, and then based on unsupervised clustering and / or feature detection and key point extraction of size parameters, it can more accurately identify and locate cell events and distinguish target events from noise events. The trajectory tracking process provides continuous motion information of cell events. The entire method achieves high-precision and high-efficiency labeling of cell events, reduces manual intervention, and improves data processing efficiency.

[0032] In some preferred embodiments, in step S2, the noise filtering process is performed based on a single or multiple cascaded event noise filters, where the event noise filter includes a background noise filter and a refractory period filter.

[0033] Specifically, the event noise filter is used to filter the noise in the event data. The noise filtering process can adopt a background noise filter, a refractory period filter or other event noise filters, or a cascade form of them. In the embodiment of the present application, the noise filtering process is preferably based on a cascaded background noise filter and a refractory period filter.

[0034] More specifically, a background noise filter refers to a technical means for identifying and removing background-related interference events in event data. It can be implemented using a method based on spatial local density analysis. For example, by counting the number of events within a certain spatial neighborhood for each event and comparing it with a preset threshold, events below the threshold are considered background noise. A refractory period filter refers to a technical means for identifying and removing pseudo-events or noise that may be generated when an event camera sensor enters a refractory period after detecting an event. It can be implemented using a method based on temporal local density analysis. For example, by checking whether other events occur at the same pixel or adjacent pixels within a very short time window after an event occurs. If so, the subsequent event may be refractory period noise and is filtered out. Cascading refers to connecting two or more filters in series, with the output of the previous filter serving as the input of the next filter to achieve step-by-step filtering.

[0035] Through the above design, the method of the present application can effectively remove noise from different sources in event data, including background noise and noise caused by the refractory period effect, thereby improving the thoroughness of noise filtering, thereby improving the quality of the effective event set, providing a purer data foundation for subsequent key point extraction and trajectory tracking, and improving the accuracy and reliability of the entire event data labeling method.

[0036] More specifically, the background noise filter has a first filtering time window, and the refractory period filter has a second filtering time window. In the background noise filter, if no other events exist within a 3×3 neighborhood of an event within the first filtering time window, the event is considered noise and filtered out by the background noise filter. In the refractory period filter, if repeated events occur at a certain location within the second filtering time window Δ, the event is considered a sudden event and filtered out by the refractory period filter.

[0037] Preferably, the first filtering time window and the second filtering time window are both set based on the aforementioned time window; wherein the first filtering time window is the same as the time window, and the second filtering time window is half of the time window.

[0038] In some preferred embodiments, in step S3, the size parameter includes a distance boundary, and the process of extracting key point information of each valid event set according to the preset size parameter based on unsupervised clustering includes: S31. Clustering the events in the valid event set based on an unsupervised clustering algorithm to obtain a clustering result containing several event clusters; S32, filtering event clusters whose event quantity in each clustering result is less than a preset second quantity threshold; S33, filtering outlier events for the remaining event clusters based on the distance boundary; S34. Output key point information based on the event cluster filtered out of the outlier events.

[0039] Specifically, an unsupervised clustering algorithm refers to an algorithm that divides data into different groups by analyzing the intrinsic structure of the data without knowing the category labels in advance. It can be implemented using algorithms such as DBSCAN, K-Means, and Mean Shift. The effective event set refers to a window event set that has undergone preliminary noise filtering and contains a sufficient number of events. It is the basis for subsequent processing. An event cluster refers to a set formed after the clustering algorithm groups event points that are adjacent in space and time. The preset second quantity threshold is a quantity standard for determining whether an event cluster is a noise cluster. Its setting can be based on empirical estimation or statistical analysis of the size of the noise event cluster. The distance boundary is a distance standard for determining whether an event in a cluster is an outlier. Its setting takes into account the reference size S and is used to adapt to targets of different sizes. In an embodiment of the present application, the second quantity threshold is preferably set to 20.

[0040] Specifically, the working principle of the above steps is as follows: First, the events in the valid event set are clustered based on an unsupervised clustering algorithm, organizing discrete event points into clusters with structural significance to form preliminary clustering results. These clusters may correspond to potential cellular events or noise. Next, the noise clusters in the clustering results are filtered, and event clusters with fewer events than the threshold are filtered out using a preset second number threshold. This is because the number of events generated by noise events or non-target objects is generally small, and the clusters formed are also relatively small. By filtering these small clusters, most of the noise interference can be effectively removed, retaining clusters that are more likely to be true cellular events. Furthermore, the remaining event clusters are filtered for outliers. Even the retained clusters may contain some outlier event points far from the cluster center. By traversing and comparing the distances of each event point within the cluster with the corresponding event cluster center based on a preset distance boundary, event points with distances greater than the distance boundary are identified as outliers and filtered out. This effectively eliminates outliers within the cluster, making the remaining event points more representative of the core area of ​​cellular events. Finally, key point information is output based on the event clusters after noise cluster filtering and outlier event filtering. Because these clusters have already removed most of the noise and outliers, the keypoint information calculated based on them will be more accurate and reliable, providing high-quality input data for subsequent trajectory tracking and cell event labeling. This process works synergistically with the previous noise filtering step to further purify the data and provide more reliable input for subsequent trajectory tracking.

[0041] Preferably, the distance boundary setting formula is as follows: B=(S / 10+1)*10 (1) Among them, B is the distance boundary and S is the reference size.

[0042] It should be noted that if the unsupervised clustering algorithm adopts the Mean Shift algorithm, the hyperparameter bandwidth in the algorithm can be directly set to the value of the above distance boundary.

[0043] In some preferred embodiments, in step S3, the size parameters include a minimum circle radius, a maximum circle radius, and a minimum distance between circle centers. Based on feature detection, the process of extracting key point information of each valid event set according to the preset size parameters includes: S31′, converting the events in each valid event set into a two-dimensional edge graph, and obtaining the corresponding event timestamps; S32′, checking the edge map based on the Hough circle detection according to the minimum circle radius, the maximum circle radius and the minimum circle center distance, so as to mark the cell events based on the circles; S33', output key point information based on circular cell events and event timestamps.

[0044] Specifically, step S31' preferably converts and obtains an edge map by reconstructing the light structure information of events in a spatiotemporal neighborhood. Reconstructing the light structure information of events in a spatiotemporal neighborhood involves converting discrete, asynchronous event data into a continuous or semi-continuous representation that reflects local brightness variations or edge features, taking into account the time and spatial location of the events and information about neighboring events. This can be achieved using techniques such as image frame generation based on event accumulation, surface fitting based on event streams, or computation of feature descriptors based on local event activity. Converting events into a two-dimensional edge map involves further processing the reconstructed light structure information to generate a two-dimensional image representation, where pixel values ​​reflect the presence of edges or regions of dramatic brightness variations at corresponding spatial locations. This can be achieved by applying edge detection operators such as Sobel, Prewitt, or Canny to the reconstructed light structure information. Hough circle detection is a feature extraction technique for detecting specific shapes (here, circles) in an image. Its principle is to map points in image space to parameter space and identify geometric structures in the image space that conform to the specific shape by searching for cumulative peaks in the parameter space.

[0045] Specifically, the above processing first converts event data into a two-dimensional edge map, enabling the application of established image processing techniques. This conversion process considers the event's spatiotemporal neighborhood information, transforming discrete events into edge representations with a certain degree of spatial continuity, laying the foundation for subsequent shape detection while preserving the timestamp information of the original event. After obtaining the edge map, the Hough circle detection algorithm actively searches for circular structures that meet preset size ranges (minimum circle radius, maximum circle radius) and spatial distribution (minimum distance between circle centers). These parameters are set based on a pre-calibrated baseline size and constrain the detection results to conform to the expected characteristics of cellular events. Hough circle detection directly exploits the circular shape characteristics that cellular events may exhibit, complementing clustering methods that rely on spatial density. The detected circular annotations represent the spatial location and size of the cellular events. Finally, these detected circular annotations are associated with the corresponding event timestamps, outputting keypoint information containing attributes such as spatial location, size, and time.

[0046] In this way, the method of the present application can more accurately identify and locate cell events with circular features, and is particularly suitable for scenarios where cell events appear as clear circular outlines in event data. This shape-based detection method, combined with or as an alternative to spatial density-based clustering methods, enhances the keypoint extraction ability for specific shape targets, improves the accuracy and robustness of keypoint extraction, and thus provides more reliable input for subsequent trajectory tracking.

[0047] Preferably, the minimum circle radius setting formula is as follows: R min =max(5*(S / 10-1),10) (2) The formula for setting the maximum circle radius is as follows: R max = 5*(S / 10+2) (3) Dist min =R max *2 / 3 (4) Among them, R min is the minimum circle radius, R max is the maximum circle radius, Dist min is the minimum center distance.

[0048] It should be noted that the edge map is actually a two-dimensional histogram of events, and its conversion process is based on the normalized weighted value of the number of all trigger events in the valid event set corresponding to each pixel statistical time window and the corresponding image pixel maximum value.

[0049] In some preferred embodiments, the following steps are further included between step S31' and step S32': S3A', downsampling the edge map.

[0050] Specifically, the downsampling process reduces the number of pixels in the edge map, thus reducing the data volume and resolution. The edge map after downsampling is then used for Hough circle detection. By performing Hough circle detection on a lower resolution edge map, the required computational resources and time are significantly reduced, improving processing efficiency. In addition, the downsampling process usually has a smoothing effect, which helps to suppress fine noise and irregularities in the edge map, making the Hough circle detection less sensitive to noise, improving the robustness and accuracy of detection, and enabling the circular structure of the cell event to be more stably identified.

[0051] In some preferred embodiments, the keypoint information includes centroid coordinates, pixel size, timestamp, and event weight.

[0052] Specifically, the centroid coordinates refer to the center position of the cell event on the image plane, which can be represented by a pixel coordinate pair (x, y). The pixel size refers to the size occupied by the cell event on the image plane, which can be represented by equivalent diameter or bounding box size, etc. The timestamp refers to the time point at which the cell event occurs, which can be represented by the microsecond or nanosecond time value output by the event camera. The event weight refers to the number of events related to the keypoint or its activity level, which can be represented by the original event count constituting the keypoint or a value calculated based on the event activity intensity.

[0053] More specifically, the method of the present application provides a more comprehensive and accurate description of the cell event by including centroid coordinates, pixel size, timestamp, and event weight in the keypoint information.

[0054] More specifically, the centroid coordinates provide the spatial position basis of the cell event and are the starting point for trajectory tracking. The pixel size complements the size characteristics of the cell event, helping to distinguish different sizes of targets or identify size changes of the same target, which is important for handling cell deformation or distinguishing similar targets. The timestamp takes advantage of the high temporal resolution characteristics of the event camera to provide a basis for accurate time correlation and speed calculation, especially suitable for trajectory reconstruction in high-speed flow scenarios. The event weight provides reliability or importance information of the keypoint, which can be used for weighted processing or as a confidence reference in the trajectory tracking process, enhancing the ability to handle noisy or incomplete data.

[0055] It should be noted that in the processing mode based on unsupervised clustering to obtain keypoint information, the centroid coordinates are the centroid coordinates of the event cluster, the pixel size can be represented by length and width or circle size (radius or diameter), the timestamp can be represented by the latest timestamp of the events in the cluster, and the event weight is the amount of data in the cluster.

[0056] It should be noted that in the processing method of obtaining key point information based on feature detection, the centroid coordinates are the coordinates of the center of the circle obtained by detection, the pixel size is the circle size (radius or diameter), the timestamp can use the event timestamp extracted previously, and the weight w is the number of votes for the center of the circle.

[0057] It should be noted that unsupervised clustering and feature detection can be used separately or in parallel, and parallel use can play a role in information complementarity.

[0058] In some preferred embodiments, step S4 includes: S41, grouping the key point information according to the pixel distance and time difference between the key point information, and assigning a trajectory ID to obtain preliminary trajectory information; S42 : For each preliminary trajectory information, merge key point information whose time difference is less than a preset first time threshold to obtain trajectory information to mark the event data stream.

[0059] Specifically, pixel distance refers to the spatial distance between two key points on the image plane. Time difference refers to the difference between the timestamps corresponding to two key points.

[0060] Specifically, step S41 obtains preliminary trajectory information by grouping keypoint information based on pixel distance and time difference between them and assigning track IDs. This step exploits the spatial and temporal correlation of keypoints in event camera data. Cellular events typically appear as spatially close keypoints on the image plane at consecutive timestamps. By simultaneously considering pixel distance and time difference, keypoints belonging to the same potential trajectory can be effectively clustered together, and these preliminary trajectory segments are assigned unique track IDs, laying the foundation for subsequent trajectory refinement. Next, step S42 merges keypoint information within each preliminary trajectory with a time difference less than a preset first time threshold to obtain the final trajectory information. Step S41 may generate multiple short track segments belonging to the same cell due to the intermittent nature of keypoint detection or the complexity of cell movement. Step S42 sets a time threshold to merge preliminary trajectory segments that are temporally consecutive (with a time difference less than the threshold). This merging process can connect originally scattered trajectory segments to form a more complete and longer trajectory, thereby improving the accuracy and continuity of trajectory tracking, and ultimately generating accurate trajectory information for event data labeling.

[0061] In some implementations, step S41 may be implemented in the following manner: Given input keypoint information, if the historical trajectory is empty, a new trajectory is created and assigned a trajectory ID, with the input keypoint as the starting point. If the historical trajectory is not empty, the input keypoint information is used to match each historical trajectory. If a match is successful, the keypoint information is associated with the corresponding trajectory to gradually form preliminary trajectory information. The matching process between keypoint information and historical trajectory is as follows.

[0062] The historical trajectory is traversed, and the tail key points of each historical trajectory are extracted. The pixel distance and time difference between the key point information and each tail key point are calculated. If the time difference is less than a preset second time threshold, and the pixel distance is less than the maximum size of all key points in the corresponding historical trajectory, the historical trajectory is considered a candidate trajectory. Finally, the candidate trajectory corresponding to the minimum pixel distance is selected as the matching result for the key point information, and the key point information is then incorporated into the historical trajectory. If no candidate trajectory exists during the matching process, a new trajectory is created, and a trajectory ID is assigned to the key point information, thus forming a new historical trajectory, with the key point as the starting point of the historical trajectory.

[0063] Therefore, the above processing method gradually incorporates each key point information into different trajectories. After completing the matching of all key point information, these assigned historical trajectories are determined as preliminary trajectory information.

[0064] In some implementations, in step S42, merging key point information is performed based on the following formula: (5) (6) (7) (8) Among them, c1 and c2 are the centroid coordinates of the two key point information to be merged, w1 and w2 are the event weights of the two key point information to be merged, t1 and t2 are the timestamps of the two key point information to be merged, s1 and s2 are the pixel sizes of the two key point information to be merged, c new 、w new , t new and s new They are respectively the centroid coordinates, event weight, timestamp and pixel size of the merged key point information.

[0065] In some preferred embodiments, the first time threshold and the second time threshold are preferably set based on a time window, respectively satisfying: T1=α·△T, 0<α<1 (9) T2=β·△T, β>1 (10) Wherein, T1 is the first time threshold, T2 is the second time threshold, AT is the time window, and a and β are preset time coefficients, which can be set according to the use requirements, such as a set to 0.5 and β set to 15.

[0066] In some preferred embodiments, the calibration process of the time window and the reference size includes: A1, acquiring the event data stream for testing, and marking the event data stream for testing based on the preset initial time window and the initial reference size to obtain pre-check trajectory information; A2, calculating the average speed information of the cell event according to the pre-check trajectory information; A3, calculating the calibration time window according to the ratio of the set expected pixel moving speed of the cell event and the average speed information; A4, calculating the average size information corresponding to the cell event as the reference size according to the pre-check trajectory information.

[0067] Specifically, the event data stream for testing refers to a section of event data used for parameter calibration, which can be a section intercepted from the complete event data stream or data specially collected for testing. The preset initial time window refers to a preliminary set time window value used when performing preliminary marking, which can be an empirical value or a randomly set value. The preset initial reference size refers to a preliminary set reference size value used when performing preliminary marking, which can be an empirical value or a randomly set value. The pre-check trajectory information refers to the trajectory data obtained through the preliminary marking process, which contains the motion trajectory, size and other information of the cell event identified under the initial parameters. The average speed information of the cell event refers to the average moving speed of the cell on the pixel plane calculated according to the pre-check trajectory information. The set expected pixel moving speed of the cell event refers to a target speed value set by the user or the system according to the actual application requirements, representing the pixel distance that the cell is expected to move within each time window. The average size information corresponding to the cell event refers to the average size of the cell on the pixel plane calculated according to the pre-check trajectory information.

[0068] More specifically, step A1 obtains a segment of test event data stream and performs preliminary labeling on this data using a preset initial time window and initial reference size. The resulting pre-flight trajectory information contains the motion trajectory and key point information of the cell events under the initial parameters, providing a data source for subsequent parameter adjustments. Step A2 estimates the average cell movement speed on the pixel plane by analyzing the temporal changes of the key points in the pre-flight trajectory. This average movement speed reflects the actual motion characteristics of the cell within the event camera's field of view under the current initial parameters and serves as a key basis for calibrating the time window. Step A3 calculates and calibrates the final time window based on the ratio of the expected pixel movement speed of the cell event to the average speed information calculated in step A2. By comparing the actual measured average speed with the expected speed and adjusting the time window size based on the proportional relationship between them, the calibrated time window can better adapt to the actual cell movement speed, ensuring that an appropriate amount of cell event information can be captured within a time window, thereby optimizing the segmentation of the event data stream. Step A4 calculates the average size information corresponding to the cell events based on the pre-check trajectory information obtained in step A1, and uses this average size information as the final reference size. The key points in the pre-check trajectory information may contain the size information of the cell events. By statistically averaging this size information, the average pixel size of the cells can be obtained. Using this average size as the reference size can provide a data-driven basis for setting size parameters in the subsequent key point extraction process, allowing the key point extraction process to more accurately identify cell events with this average size feature, thereby improving the accuracy of key point extraction.

[0069] Through the above scheme, the method of the present application realizes the adaptive calibration of the time window and the reference size, improves the degree of automation of the event data marking method and its adaptability to different cell types and motion states, and adjusts key parameters by using the feedback of the preliminary marking results so that the parameter settings can more accurately reflect the actual cell movement speed and size characteristics, thereby improving the marking accuracy and robustness, and better leveraging the advantages of the high temporal resolution of the event camera.

[0070] In some preferred embodiments, the pre-check trajectory information is composed based on a plurality of pre-check key point information; Step A2 includes: A21, extracting preflight trajectory information having a number of preflight key point information greater than a preset third number threshold, and calculating average speed information of the preflight key point information based on the preflight trajectory information; Step A4 includes: A41 . Extract preflight track information with a number of preflight key point information greater than a third number threshold, and calculate average size information of the preflight key point information based on the preflight track information as a reference size.

[0071] Specifically, the third quantity threshold refers to the minimum number of key points used to screen pre-inspection trajectory information. This threshold is set to distinguish reliable trajectories that contain sufficient information to represent the true motion and size characteristics of cell events from unreliable trajectories with too few key points due to noise, occlusion, or transient appearance. This threshold can be set based on the actual application scenario, the characteristics of the event camera, and the expected noise level; in this embodiment of the application, the third quantity threshold can be set to 5.

[0072] Specifically, the above embodiment introduces a screening step: when calculating the average velocity information of a cell event, instead of directly using all acquired preflight trajectory information, the number of preflight keypoint information contained in each preflight trajectory information is first checked. Only when the number of keypoints contained in a preflight trajectory information exceeds a preset third threshold value is this trajectory retained for calculation. Based on these screened preflight trajectory information with a sufficient number of keypoints, the average velocity information of the preflight keypoint information is calculated. Similarly, when calculating the average size information corresponding to a cell event and using it as a reference size, the same screening strategy is employed. Preflight trajectory information with a number of preflight keypoint information exceeding the preset third threshold value is similarly extracted, and the average size information of the preflight keypoint information is calculated based on these screened trajectory information. In this way, unreliable trajectories containing a small number of keypoints, which may be due to incomplete data or noise interference, are eliminated from interfering with the average velocity and average size calculations. This screening process ensures that the average velocity and average size data used to calibrate the time window and reference size are more accurate and reliable.

[0073] Through the above processing, the method of the present application avoids the interference of unreliable trajectories containing a small number of key points on the calculation of average speed and average size. As a result, the calculated average speed and average size can more accurately reflect the true characteristics of cellular events, making the calibration of time windows and reference sizes more precise, and improving the accuracy of subsequent event data labeling.

[0074] In some preferred embodiments, in step A2, the average speed information can be calculated by extracting key point information with a certain number of intervals in the pre-check trajectory information, such as extracting key point information from each pre-check trajectory information with 3 key points as intervals and extracting corresponding coordinates and timestamps to calculate the average speed information.

[0075] In some preferred embodiments, in step A3, the time window is calculated as follows: (11) Where D is the expected pixel movement speed of the cell event, is the average speed information.

[0076] In some preferred embodiments, in step A4, for the circle size, the average size information may be the diameter; for the rectangle size, the average size information may be the larger value of the length and width.

[0077] In some preferred embodiments, step A1 uses the event data stream for testing as the event data stream to be processed, the initial time window as the time window, and the initial reference size as the reference size, and executes the trajectory information obtained by steps S1-S4 as pre-inspection trajectory information.

[0078] Specifically, executing steps S1-S4 involves applying a specific event data labeling method, which includes steps such as segmenting the event data stream, filtering noise, extracting keypoint information, tracking trajectories, and configuring trajectory IDs. Pre-flight trajectory information refers to the trajectory data obtained by executing this event data labeling method based on the test event data stream and preset initial parameters (initial time window and initial reference size). This data is used for subsequent calibration calculations.

[0079] Specifically, the design utilizes the event data labeling method itself, which will be used in actual applications, to generate pre-flight trajectory information for calibration. First, the event data stream used for testing and calibration is used as input to the labeling method. Then, when executing each step of the labeling method, a preset initial time window is used as the time window parameter, and a preset initial reference size is used as the reference size parameter. Next, all steps of the labeling method are fully executed, including segmenting the event data stream based on the time window, performing noise filtering on the segmented event set, extracting key point information based on the reference size, tracking the key point information and assigning trajectory IDs, and finally outputting the trajectory information. Finally, the trajectory information obtained in this manner is designated as the pre-flight trajectory information for calibration. By using the same labeling process as in actual applications to generate the pre-flight trajectory information, the calibration data is ensured to truly reflect the behavior and characteristics of the labeling method when processing event data. Based on this highly consistent pre-flight trajectory information, subsequent calculations of average speed and average size yield more accurate statistical results. This makes the final calibrated time window and baseline size parameters more suitable for optimizing the performance of the event data labeling method in practical applications, improving the reliability and effectiveness of the calibration results. This approach tightly couples the calibration data generation process with the method to be calibrated, enabling the calibration results to more accurately reflect the method's performance in real-world scenarios, thereby enhancing the applicability and accuracy of the calibration parameters.

[0080] In the above embodiment, step A1 can be performed in the following manner: B1. Obtain a test event data stream, and segment the test event data stream based on a preset initial time window to generate a test window event set; B2. Perform noise filtering on each test window event set, and retain the test window event sets with a number of remaining events greater than a preset first threshold value to form a valid pre-check event set; B3. Based on unsupervised clustering, extract key point information of each valid pre-check event set according to the reference size parameters set based on the initial benchmark size; B4. Track the key point information of the valid pre-check event set and configure the track ID to generate pre-check track information.

[0081] In some embodiments, the reference size parameter used in step B3 may also be replaced by a parameter set based on the frame size, such as setting the target object (cell event to be measured) to occupy 1 / 8 of the shortest side of the frame.

[0082] In some embodiments, after the trajectory information is generated in step S4, the method of the present application can archive the key point information by event marking, such as appropriately enlarging the key point size based on the centroid coordinates of the key point information to form a region of interest (ROI), and marking all events covered by the ROI with the key point label labelID. Secondly, all events involved in the formation of the key point information are marked with the key point label labelID, and the value on the labelID dimension is the trajectory ID. Different trajectory IDs correspond to different target objects (cell microspheres).

[0083] Second, please refer to Figure 3 Some embodiments of the present application further provide an event data marking system based on neuromorphic vision, for marking cell events collected by an event camera, the system comprising: A preprocessing module 201 is configured to obtain an event data stream to be processed, segment the event data stream based on pre-calibrated time windows, generate window event sets to be processed, perform noise filtering on each window event set, and retain window events with a number of remaining events greater than a preset first threshold value to form a valid event set; A key point detection module 202 is configured to extract key point information of each valid event set based on unsupervised clustering and / or feature detection according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size; The key point tracking module 203 is configured to track the key point information and configure a track ID, thereby generating track information to mark the event data stream.

[0084] The system of the present application can fully utilize the high temporal precision characteristics of the event camera to realize cell labeling, avoiding the temporal resolution loss caused by the traditional image-assisted labeling method. It effectively removes the noise in the event data through noise filtering, improves the purity of the labeling results, and then based on unsupervised clustering and / or feature detection and key point extraction of size parameters, it can more accurately identify and locate cell events and distinguish target events from noise events. The trajectory tracking process provides continuous motion information of cell events. The entire system achieves high-precision and high-efficiency labeling of cell events, reduces manual intervention, and improves data processing efficiency.

[0085] In some embodiments, the system further includes a marking module for archiving key point information by means of event marking.

[0086] Thirdly, please refer to Figure 4 Some embodiments of the present application also provide a structural diagram of an electronic device. The present application provides an electronic device, including: a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other through a communication bus 303 and / or other forms of connection mechanisms (not marked). The memory 302 stores computer-readable instructions executable by the processor 301. When the electronic device is running, the processor 301 executes the computer-readable instructions to execute the method in any optional implementation of the above embodiments.

[0087] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the method of any optional implementation of the above embodiment is executed. The computer-readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk, or optical disk.

[0088] In the embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interface, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0089] In addition, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0090] Furthermore, the functional modules in each embodiment of the present application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0091] In this document, relational terms such as first and second, etc. are used merely to distinguish one entity or operation from another entity or operation, but do not necessarily require or imply any actual relationship or order between these entities or operations.

[0092] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. For those skilled in the art, various modifications and variations of the present application are possible. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for marking event data based on neuromorphic vision, for marking cell events collected by an event camera, characterized in that: The method comprises the following steps: S1. Obtain an event data stream to be processed, and segment the event data stream based on a pre-calibrated time window to generate a window event set to be processed; S2. Perform noise filtering on each window event set, and retain window events whose number of remaining events is greater than a preset first number threshold to form a valid event set; S3. Extracting key point information of each valid event set based on unsupervised clustering and / or feature detection according to preset size parameters, where the size parameters are set based on a pre-calibrated reference size; S4. Track the key point information and configure a trajectory ID to generate trajectory information to mark the event data stream.

2. The event data labeling method based on neuromorphic vision according to claim 1, characterized in that: In step S2 , noise filtering is performed based on a single or multiple cascaded event noise filters, where the event noise filters include a background noise filter and a refractory period filter.

3. The event data marking method based on neuromorphic vision according to claim 1, characterized in that: In step S3, the size parameter includes a distance boundary. Based on unsupervised clustering, the process of extracting key point information of each valid event set according to the preset size parameter includes: S31. Clustering the events in the valid event set based on an unsupervised clustering algorithm to obtain a clustering result containing several event clusters; S32, filtering event clusters whose event quantity in each clustering result is less than a preset second quantity threshold; S33, filtering outlier events from the remaining event clusters based on the distance boundary; S34. Output key point information based on the event cluster filtered out of the outlier events.

4. The event data marking method based on neuromorphic vision according to claim 1, characterized in that: In step S3, the size parameters include the minimum circle radius, the maximum circle radius, and the minimum circle center distance. Based on feature detection, the process of extracting key point information of each valid event set according to the preset size parameters includes: S31′, converting the events in each valid event set into a two-dimensional edge graph, and obtaining the corresponding event timestamps; S32′, checking the edge map based on the Hough circle detection according to the minimum circle radius, the maximum circle radius and the minimum circle center distance, so as to mark cell events based on circles; S33': output the key point information based on the circular cell event and the event timestamp.

5. The event data marking method based on neuromorphic vision according to claim 4 is characterized in that: The following steps are included between step S31' and step S32': S3A′, performing downsampling processing on the edge map.

6. The event data marking method based on neuromorphic vision according to claim 1, characterized in that: Step S4 includes: S41, grouping the key point information according to the pixel distance and time difference between the key point information, and assigning a trajectory ID to obtain preliminary trajectory information; S42 : For each preliminary trajectory information, merge key point information whose time difference is less than a preset first time threshold to obtain the trajectory information to mark the event data stream.

7. The event data marking method based on neuromorphic vision according to claim 1, characterized in that: The calibration process of the time window and the reference size includes: A1. Acquire a test event data stream, and mark the test event data stream based on a preset initial time window and an initial reference size to obtain pre-check trajectory information; A2. calculating average velocity information of cell events based on the pre-detection trajectory information; A3. calibrating the time window according to the ratio of the expected pixel movement speed of the set cell event and the average speed information; A4. Calculate average size information corresponding to cell events according to the pre-detection trajectory information, and use the average size information as the reference size.

8. An event data marking system based on neuromorphic vision for marking cell events collected by an event camera, characterized in that: The system comprises: a preprocessing module, configured to obtain an event data stream to be processed, segment the event data stream based on pre-calibrated time windows, generate window event sets to be processed, perform noise filtering on each window event set, and retain window events whose number of remaining events is greater than a preset first threshold value to constitute a valid event set; A key point detection module, configured to extract key point information of each valid event set based on unsupervised clustering and / or feature detection according to preset size parameters, wherein the size parameters are set based on a pre-calibrated reference size; The key point tracking module is used to track the key point information and configure a track ID, and generate track information to mark the event data stream.

9. An electronic device, characterized in that: The method comprises a processor and a memory, wherein the memory stores computer-readable instructions. When the computer-readable instructions are executed by the processor, the steps in the method according to any one of claims 1 to 7 are executed.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are executed.

Citation Information

Patent Citations

  • Systems and methods for tracking single-cell movement trajectories

    US20200219266A1

  • Product correlation analysis using deep learning

    WO2020023799A1