Method and system for adaptive corner detection using dynamic vision sensor

Abstract

A method for adaptive corner detection is provided. The method includes: obtaining one or more event data from a dynamic vision sensor; capturing a plurality of recorded events of the event data and organizing them into a 2D array; transforming the 2D array into one or more ordered surface (OS) matrices by populating one or more empty image matrices with the recorded events based on coordinates of the recorded events and assigning order values; and applying a corner detector to the OS matrix. A system for adaptive corner detection is also provided.

Description

Technical Field

[0001] The present invention generally relates to machine vision technology. More specifically, the present invention relates to corner detection methods and systems. Background Art

[0002] Machine vision is crucial for technological advancements in many fields, supporting autonomous vehicles, drones, and robotics. Conventional cameras capture scenes via fixed exposure, facing challenges in terms of large amounts of data and high transmission speed requirements. Event cameras, which address this by sampling only when there are temporal contrast changes, offer benefits such as higher dynamic range and lower latency.

[0003] In machine vision, it is important to have clear and easily transferable features that work well in various situations and for different tasks. One of the most commonly used and informative features to detect is the "corner point". For resource-constrained edge Internet of Things (IoT) applications, it is crucial to efficiently process event data for tasks such as corner detection, and there are challenges in current edge IoT applications.

[0004] Edge IoT applications require efficient processing methods. The sparse event output of a stationary camera enables deep neural networks (DNNs) in edge IoT via a work-loop processor. Some state-of-the-art algorithms show promise for low-level processing such as noise removal, but corner detection methods lack optimization for extreme edge devices. The challenge is to develop an energy-efficient corner detection technology for extreme edge applications.

[0005] Corners are often detected using methods such as the Harris corner detector or those based on segments (such as features from the Features from Accelerated Segment Test (FAST)). These methods are used for event-based data using the "Surface of Active Events" (SAE) data structure.

[0006] Corner detection is crucial for computer vision tasks, and existing technologies face challenges in throughput and accuracy. Harris-based corner detection methods (such as eHarris) apply the Harris detector to binary SAE data segments around each event. However, the processing speed is limited by the number of events, making it less suitable for real-world applications. Segment-based methods like FAST process SAE data faster but may sacrifice accuracy. Asynchronous Corner Detection (ACD) addresses these challenges by enhancing throughput and accuracy. However, due to per-event updates, it has difficulties in energy efficiency, which hinders its adoption in extreme-edge application-specific integrated circuits (ASICs). LuvHarris aims to enhance the processing speed while maintaining accuracy by introducing a "Threshold-Ordinance Surface (TOS)" data structure and implementing a multi-threaded processing pipeline. Despite achieving advanced accuracy and increased throughput, LuvHarris faces challenges in energy consumption during TOS updates, which limits its direct implementation on low-power edge devices such as application-specific integrated circuits (ASICs). Therefore, an effective solution for corner detection in resource-constrained environments is needed. Summary of the Invention

[0007] An object of the present invention is to provide a method and system that solve the foregoing drawbacks and unmet needs in the prior art. The present invention provides an adaptive corner detection method that can be deployed on an edge device, and a system that can be used as an edge device. The advantages of the method and system include: a way to maintain the temporal order of events without using too much memory to store events; an effective method to transform events into a fixed-size matrix for corner detection; and a way to quickly identify corners in events without loss of accuracy.

[0008] According to a first aspect of the present invention, there is provided a method for adaptive corner detection. The method includes: obtaining event data from a dynamic vision sensor; capturing a plurality of recorded events of the event data and organizing them into a two-dimensional (2D) array; transforming the 2D array into one or more Ordered Surface (OS) matrices by filling one or more empty ImageMatrices (IM) with the recorded events based on the coordinates of the recorded events and assigning order values; and applying a corner detector to the OS matrix.

[0009] According to a second aspect of the present invention, a system for adaptive corner detection is provided. The system includes a dynamic vision sensor and a machine vision processor electrically connected to the dynamic vision sensor. The machine vision processor is configured to obtain event data from the dynamic vision sensor; capture a plurality of recorded events of the event data and organize them into a 2D array; transform the 2D array into one or more OS matrices by filling one or more empty IMs with the recorded events based on the coordinates of the recorded events and assigning order values; and apply a corner detector to the OS matrix to output a corner detection result.

[0010] According to an embodiment of the present invention, the recorded events are arranged in a globally time-ordered manner in the 2D array.

[0011] According to another embodiment, the recorded events are arranged in multiple rows in the 2D array. Each of the rows contains a series of recorded events arranged in a globally time-ordered manner. The number of recorded events in each row is the same, and the rows are arranged in a globally time-ordered manner.

[0012] According to another embodiment, the maximum number of recorded events in each row is in the range of 25 to 100.

[0013] According to another embodiment, each OS matrix is derived from all the rows of the recorded events in the 2D array.

[0014] According to another embodiment, the step of applying a corner detector to one of the OS matrices includes: extracting an image patch of the elements in the OS matrix; applying a Harris detector to the image patch to generate a Harris score; and comparing the Harris score generated by the Harris detector with a predefined threshold.

[0015] According to another embodiment, the method includes: performing permutation normalization on the image patch of the elements before applying the Harris detector.

[0016] According to another embodiment, the image patch has M rows and N columns, and M is in the range of 7 to 11, and N is in the range of 7 to 11.

[0017] According to another embodiment, the method includes: applying a spatio-temporal correlation filter to the event data before organizing the recorded events into a structured 2D array.

[0018] In summary, the method and system for adaptive corner detection according to the embodiments of the present invention achieve effective corner detection. Data from an event camera can be processed at high speed, and the method and system maintain the reliability of corner detection. In addition, the method and system of the embodiments process data with low power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Embodiments of the present invention are described in more detail below with reference to the drawings, in which:

[0020] Figure 1 Schematic diagram depicting a 2D array according to an embodiment of the present invention;

[0021] Figure 2 Schematic diagram depicting a 2D array and a corresponding OS according to another embodiment of the present invention;

[0022] Figure 3 Schematic diagram depicting a system for adjusting corner detection according to another embodiment of the present invention;

[0023] Figure 4 Flowchart of a program for a method or system for adjusting corner detection according to some embodiments of the present invention;

[0024] Figure 5 Flowchart of a program for a method or system for adjusting corner detection according to another embodiment of the present invention;

[0025] Figure 6 Schematic diagram depicting a 2D array according to another embodiment of the present invention;

[0026] Figure 7 Algorithm for performing a transformation program according to an embodiment of the present invention;

[0027] Figure 8 Flowchart of a program for a method or system for adjusting corner detection according to another embodiment of the present invention;

[0028] Figure 9 Flowchart of a program for a method or system for adjusting corner detection according to yet another embodiment of the present invention; and

[0029] Figure 10 Algorithm for performing local image block construction and Harris calculation according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] In the following description, methods and systems for adaptive corner detection etc. are presented as preferred examples. It will be apparent to those skilled in the art that modifications including additions and / or substitutions can be made without departing from the scope and spirit of the present invention. Specific details may be omitted so as not to obscure the present invention; however, this disclosure is written to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0031] According to a first aspect of the present invention, there is provided a method for adaptive corner detection using a dynamic vision sensor. The method proposes a corner detection pipeline that can be deployed on an edge device. The features of the method include: saving events without using too much memory while maintaining the time order of the events; effectively converting the events into a fixed-size matrix for corner detection; and quickly identifying corners in the events without loss of accuracy.

[0032] The method includes the following procedures: updating a 2D array with time sorting; constructing an Ordered Surface (OS) matrix that introduces relative spatial sorting; and performing corner detection.

[0033] However, the method is not limited to these procedures. A detailed description providing a full explanation of this method will be given below.

[0034] The steps of the method of this embodiment include obtaining event data from a dynamic vision sensor (DVS).

[0035] The DVS of this embodiment is a type of sensor used in computer vision and robotics to detect changes in visual information in an efficient and asynchronous manner. The DVS may include an event camera. In some other embodiments, the DVS may include a neuromorphic camera, a silicon retina, or an asynchronous camera.

[0036] An event camera is a specialized type of camera that is fundamentally different from a traditional camera. Instead of capturing images at a fixed rate, an event camera detects changes in brightness or intensity at the pixel level and transmits information only when a significant change occurs. This unique feature allows them to operate with particularly low latency and high temporal resolution, making them well-suited for real-time capture of fast and dynamic events, such as fast motion or sudden changes in lighting.

[0037] The DVS of the embodiment provides event data. The event data includes a plurality of recorded events, and each of the recorded events contains four key parameters. The recorded events generated by the event camera contain key information about dynamic changes in the visual environment. The four key parameters include: pixel coordinates (x, y), timestamp (t), and polarity (p).

[0038] The pixel coordinates precisely locate the position in the camera's field of view where the event occurs, thus providing spatial specificity. The timestamp provides the exact moment when the event occurs, allowing for accurate temporal resolution. Importantly, the polarity indicates whether the change in brightness is an increase or a decrease, thus conveying the directionality of the observed brightness change.

[0039] These parameters together contribute to the unique advantages of the event camera. The asynchronous nature of event data transmission driven by these parameters enables them to capture fast and subtle changes in real time with minimal latency. This unique way of sensing visual information makes the DVS of this embodiment particularly valuable in applications that require fast and precise responses, such as robotics, autonomous systems, and computer vision tasks where traditional frame-based cameras may be insufficient.

[0040] In this embodiment, after obtaining the event data, the method includes capturing the recorded events of the event data and organizing them into a 2D array.

[0041] In this step, at least some of the continuously recorded events are captured. In this embodiment, the recorded events are captured in real time, and the recorded events are organized according to the timestamps of the recorded events. Specifically, the pixel coordinates (i.e., x, y) are the main information of the elements stored in the 2D array, and each of the elements corresponds to one of the recorded events, while the arrangement is based on the timestamp of each recorded event.

[0042] In other words, in the 2D array of this embodiment, each element corresponds to a recorded event and contains the basic pixel coordinates, denoted as x and y. The arrangement of the elements is based on the timestamps of the recorded events, thus providing the time order of the events.

[0043] The 2D array according to the method of this embodiment is initialized with 0 as its initial entry value. Subsequently, after obtaining data from the event camera, the time series of event records is systematically filled into the corresponding 2D array entries. This process ensures that the 2D array contains the time evolution of the recorded events, thus providing a structured representation of the data in the context of data processing.

[0044] In other words, in this embodiment, after capturing multiple recorded events, the recorded events are arranged in multiple rows. During the organization of the recorded events, after all the entries in a row of entries are filled with recorded events, the subsequent recorded events are arranged to fill the entries of the next row, and so on. The entries of the rows not filled with recorded events remain containing zeros. When the multiple rows are constructed, a 2D array corresponding to the information of the recorded events is formed.

[0045] In the 2D array of this embodiment, the arrangement of the recorded events contains time information. Each of the recorded events has coordinate parameters indicating the position, especially the pixel coordinates on the coordinate system. Therefore, the 2D array contains both spatial and time information.

[0046] Figure 1It is a schematic diagram of the 2D array of this embodiment. In this embodiment, each element in the 2D array specifically contains coordinates (x, y). For example, element A(1) contains (x1, y1) and element A(C) contains (xc, yc), thus generating a small and compact array size, which is C×D to store events. C indicates the number of columns, and D indicates the depth of the array. The memory cells (i.e., elements) in the 2D array now store the coordinates of the incoming events instead of timestamps, and this design is particularly advantageous for subsequent corner detection procedures. In another embodiment, the element can be an Address Event Representation (AER) event and contain other information of the recorded events of the event data from the DVS.

[0047] In this embodiment, after organizing the 2D array, the method includes transforming the 2D array into an OS matrix by filling an empty image matrix (IM) with the recorded events based on the coordinates of the recorded events and assigning order values.

[0048] In the transformation process from the structured 2D array to the OS, we first capture the recorded events and organize them into a 2D array, where each element represents a specific event and contains pixel coordinates (x, y). Instead of storing timestamps, we use order values, which are indices indicating the temporal order of the events. This eliminates the need for large timestamp storage, thus achieving a more memory-efficient representation.

[0049] Next, the OS is constructed by filling the empty IM with the pixel coordinates of the recorded events and assigning an order value to each element. The IM effectively organizes the events into a spatial structure, thus restoring the spatial ordering that may be lost in the 2D array representation. The assignment of the order values maintains the temporal ordering of the events, thus creating a coherent and effective representation for subsequent corner detection.

[0050] Compared with traditional timestamp-based representations (such as time surfaces), this method not only reduces the memory footprint but also provides a structured OS matrix that facilitates effective corner detection methods. The resulting OS matrix organized in space and time serves as the basis for further image processing and analysis in various applications.

[0051] The following description refers to Figure 2 . An exemplary and simplified 2D array 10 is presented. The 2D array 10 has 4 rows, and each row has 4 elements. In this embodiment, the elements in a row are arranged from left to right, and the rows are arranged from top to bottom. According to the order of these elements, order values are assigned from 1 to 16 respectively, and each order value is filled in the position in the IM according to each coordinate of the element, so as to form the OS matrix 11.

[0052] Through this transformation, spatial information is reconstructed in the OS, and subsequent procedures for corner detection can be performed. In other words, by employing a 2D array and an OS matrix, we minimize the event data from the DVS. This reduction in data improves the efficiency of the subsequent corner detection process. Essentially, these steps optimize data storage without compromising the spatial and temporal information necessary for corner detection. Thus, the detector operates quickly and efficiently.

[0053] After the transformation of the OS matrix, the method of this embodiment continues to apply a corner detector to the OS matrix. Due to the reduction in data, the performance efficiency of the corner detector is improved. Thus, the method for adaptive corner detection of this embodiment can provide an effective solution for corner detection. This customized method not only improves the efficiency of corner detection but is also particularly suitable for resource-constrained environments such as IoT. Thus, the detector operates quickly and efficiently, making it an ideal solution for edge IoT applications.

[0054] In another embodiment of the present invention, a system for adaptive corner detection is provided. The system includes a DVS and a machine vision processor electrically connected to the DVS.

[0055] The machine vision processor of this embodiment obtains event data from the DVS and performs subsequent steps for corner detection. Specifically, the device captures a plurality of recorded events within the event data and organizes them into a structured 2D array. Subsequently, it transforms this 2D array into an OS matrix by filling the IM with the recorded events based on the coordinates of the recorded events and assigning order values. Finally, the machine vision processor applies a corner detector to the resulting OS matrix.

[0056] Similar to the method outlined in the previous embodiment, the adaptive corner detection system in this embodiment excels at efficiently performing corner detection with low system requirements. This feature makes the system a suitable choice for deployment in edge IoT applications.

[0057] The following description refers to Figure 3System 100 for adaptive corner detection has a DVS 110 and a machine vision processor 120. The machine vision processor 120 obtains event data 101 from the DVS 110, and the machine vision processor 120 organizes the recorded events of the event data 101 into a 2D array 102, and the machine vision processor 120 transforms the 2D array 102 into an OS matrix 103, and the machine vision processor 120 applies a corner detector to the OS matrix and generates a detection score 104, and the machine vision processor 120 can output an indicator 105 by comparing the detection score 104 with a predefined threshold. The indicator 105 can indicate whether a corner is detected in the event data 101, and the indicator 105 can be transmitted to an output device such as a display, an optical signal, etc., or proceed to a subsequent program for autonomous control of robotics or vehicles.

[0058] In this embodiment, the machine vision processor 120 may include an edge processing unit. The edge processing unit is optimized for power efficiency and can handle real-time processing. In some embodiments, the edge processing unit includes: ARM-based microcontrollers (MCUs) from STMicro (STM32F412, STM32F746, STM32H743), NXP (LPC1751FBD80), and Microchip (PIC32C series). By utilizing the adaptive corner detection procedure of this embodiment, our system excels in optimizing corner detection efficiency across various edge processing units. This makes it particularly suitable for machine vision applications and other automatic movement monitoring devices integrated into vehicles, demonstrating its versatility and effectiveness in dynamic environments.

[0059] Figure 4 Flowchart of a program for a method and system for adaptive corner detection showing some embodiments of the present invention. In some embodiments, during the process of organizing the recorded events into a 2D array (S12), the events are structured and arranged according to the global time order within the 2D array.

[0060] In these embodiments, the global time ordering ensures that the recorded events in the 2D array maintain a coherent time sequence, which is crucial in corner detection. The 2D array contains a clear and precise time order of the recorded events for corner detection.

[0061] Moreover, the global time ordering helps in the efficient processing of the recorded events in the 2D array. This enables a simplified method for corner detection, thereby improving the overall system efficiency.

[0062] In these embodiments, global time sorting helps to accurately capture and represent the temporal evolution of visual information. This precision is beneficial for corner detection, enabling the ability to detect corners with high accuracy.

[0063] According to various embodiments of the present invention, the recorded events are arranged in multiple rows in a 2D array. Each of the rows has a series of recorded events arranged in global time sorting, and the number of recorded events in each row is the same.

[0064] The rows of these embodiments are arranged in global time sorting. Specifically, the timestamps of the elements (i.e., events) arranged in one row are less than all the timestamps of the elements (i.e., events) arranged in the subsequent row, and the timestamps of the elements (i.e., events) in the second row are greater than all the timestamps of the elements (i.e., events) in the first row.

[0065] In these embodiments, rows with a consistent number of elements simplify the data processing pipeline. This consistency allows for standardized procedures and calculations when working with each row, thus simplifying the computational process steps of the system.

[0066] Also, in these embodiments, the consistent row length creates opportunities for parallelization in data processing. By making the rows have the same number of elements, parallel processing techniques can be more easily applied to GPUs and multi-core processors, potentially achieving a significant acceleration in overall event data processing. This benefit is particularly important in applications where real-time or high-throughput processing is necessary.

[0067] In one embodiment of the present invention, the maximum number of recorded events in each row is 100. However, the present invention is not limited to this number. In some other embodiments, the maximum number of recorded events in each row can be 25, 50, 100, or any value from 25 to 100. By utilizing a 2D array with these configurations, the method or system can change the frequency at which the system performs corner detection by changing C. The 2D array can also be customized by being able to shrink as the event rate of the scene decreases.

[0068] In this embodiment, the recorded events are continuously captured and organized, forming a one-dimensional sequence in multiple rows, and there are no repetitions of the recorded events in these rows. However, the present invention is not limited to this manner. The following description refers to Figure 1 . In some other embodiments of the present invention, the recorded events are organized in batches. For example, the number of recorded events in each batch is C, and the number of batches in this 2D array is D. In these embodiments, an overlapping stride is applied to the rows of the recorded events in the 2D array. In other words, some of the recorded events at the end of one batch are included at the beginning of the next batch, allowing for a more continuous and overlapping analysis of the event history.

[0069] In these embodiments, applying an overlapping stride across the rows of the recorded events within the 2D array introduces a key feature that optimizes the system's ability to capture dynamic and evolving patterns in visual data. The use of an overlapping stride enables the system or method to more comprehensively explore and analyze the temporal relationships between the recorded events. Instead of processing each row in isolation, the overlapping stride ensures that adjacent rows share a portion of their recorded events, allowing for seamless integration of temporal information across the 2D array.

[0070] The primary advantage of applying an overlapping stride in the methods or systems of these embodiments lies in its enhancement of the temporal context. By incorporating events from adjacent rows into the processing pipeline, the method or system gains a more nuanced understanding of the temporal evolution of the visual information. This is particularly beneficial in scenarios where the recorded events occur rapidly or in quick succession, ensuring that the system does not miss critical temporal correlations. The overlapping stride effectively creates a temporal buffer, enabling the system or method to capture and interpret the recorded events within a wider temporal window.

[0071] Furthermore, in these embodiments, the use of an overlapping stride aligns with the principle of adaptability in dynamic environments. It provides the system or method with the flexibility to adapt to the varying speed or rate of change of the visual input. Whether the events occur rapidly or at a slower pace, the overlapping stride enables the system or method to maintain responsiveness and accuracy in capturing the temporal dynamics. This feature enhances the robustness of the system in real-world applications, making it well-suited for corner detection in dynamic and unpredictable environments.

[0072] Figure 5 A flowchart of a program for methods and systems for adaptive corner detection demonstrating some other embodiments of the present invention. In these embodiments, the method or system applies a Spatial-Temporal Correlation Filter (STCF) to the event data (S22) before organizing the recorded events into a 2D array.

[0073] In these embodiments, applying the STCF to the recorded events before organizing them into a 2D array introduces a powerful feature that enhances the system's ability to discern meaningful patterns in the event data. This filtering mechanism operates by considering both the spatial and temporal dimensions of the recorded events, thereby capturing the correlations between adjacent events in a coherent manner.

[0074] The methods or systems of these embodiments apply STCF after obtaining event data from a DVS. The main advantage of this STCF lies in its ability to prioritize relevant information and thus remove noise or insignificant events. By considering the correlation between events in both space and time, the filter refines the event data, emphasizing sequences of events that may represent meaningful patterns or visual features. This not only improves the accuracy of subsequent processing steps but also contributes to the overall efficiency of the system by focusing computational resources on coherent information.

[0075] Furthermore, step S22 of applying STCF is in line with the principles of neuromorphic vision, mimicking the brain's ability to recognize patterns in an overall and context-aware manner. This feature ensures that the systems or methods of these embodiments not only capture individual events but also interpret their collective significance, thus facilitating more nuanced and context-related analysis. In essence, STCF acts as a sophisticated preprocessing step, enabling the system or method to have the ability to discern meaningful patterns from the recorded event stream, thereby enhancing the overall performance of the system in corner detection or any other related event-based processing.

[0076] In some embodiments of the present invention, the method and the system for processing the method include using row pointers and column pointers. The following description refers to Figure 6 . The row pointer 12 and the column pointer 13 are included in a 2D array. The row pointer and the column pointer track the insertion positions in the 2D array.

[0077] In these embodiments, the row pointer 12 and the column pointer 13 are essential elements for navigating and accessing specific elements within the 2D array that organizes the recorded events. The rows of the 2D array are logical rows implemented by the column pointer, which moves across the physical rows in memory in a cyclic manner.

[0078] The row pointer of these embodiments is a variable or indicator that tracks the current row being processed or accessed within the 2D array. As the system iterates through the rows, the row pointer is updated to point to the next row of interest. This allows the system to process events sequentially along the time dimension.

[0079] On the other hand, the column pointer of these embodiments operates within a specific row. It indicates the current position or element within the row that the system or method is examining or processing. When the recorded events are organized into columns, the column pointer helps to navigate through the spatial dimension of the 2D array.

[0080] The two pointers of these embodiments are crucial for maintaining the system's awareness of its position within the data structure, enabling it to effectively process and analyze the recorded events. They play a key role in the sequential analysis of the recorded events or when it is necessary to access and manipulate specific recorded events within the 2D array.

[0081] In other words, the row pointer and the column pointer are analogous to navigation tools that keep our system precisely tuned to the big picture of the recorded events in the architecture of our 2D array. The row pointer is responsible for our journey through time, carefully indicating the specific row under review as we traverse the time series of events. This is a guide to ensure that we always stay on track and never lose our position in the dynamic flow of time data. At the same time, the column pointer is our spatial maestro, orchestrating the movement within a given row. It directs our attention to the exact recorded events within that row, allowing us to navigate the spatial dimensions of our array meticulously. Together, these pointers provide the system or method with the flexibility to explore both time and space, thus ensuring an efficient and coherent analysis of the recorded event data.

[0082] The following description refers to Figure 6 The number of rows in the 2D array of these embodiments is (D + 1), which means that an extra row is included in this 2D array. The extra row is used to allow the insertion of recorded events when executing subsequent procedures. It allows for the dynamic insertion of events while performing the step of applying a corner detector on the OS matrix.

[0083] The extra row of these embodiments may be implemented as a buffer or a placeholder row where new events can be temporarily stored before being incorporated into the OS. This approach is particularly advantageous when dealing with asynchronous event data from a DVS where events occur at irregular intervals.

[0084] For example, in these embodiments, the step of applying a corner detector to the OS matrix involves using a Harris detector. When the DVS detects new events, they can be inserted into the extra row without interrupting the ongoing Harris detector operation on the rest of the OS matrix. This ensures a continuous flow of data processing without the need to pause or reset the analysis when new events occur.

[0085] Moreover, the ability to insert the recorded events into a separate extra row while simultaneously performing the Harris detector operation on the OS matrix helps to minimize latency. The method and the system using the method can adapt to real-time changes in visual input, thus capturing and evaluating the recorded events near real-time without significant delay.

[0086] In addition, the extra row of these embodiments provides a mechanism for efficient memory management. When deemed necessary, new events can be selectively incorporated into the OS matrix, thus preventing unnecessary memory expansion or reallocation, which is crucial in resource-constrained environments.

[0087] The following description refers to Figure 2In some embodiments, the OS matrix 11 is derived from all rows of recorded events in the 2D array 10, so there will be sufficient temporal and spatial information to perform meaningful corner detection. Transforming the full 2D array into a signal OS matrix provides several advantages in data processing.

[0088] A key benefit of the transformation of these embodiments is that space-time information is merged into a unified structure, thereby simplifying subsequent analysis. This transformation simplifies calculations, which can be more effectively applied to a single OS, rather than navigating through a complex 2D array. In addition, transitioning to a single OS matrix enhances data locality, potentially optimizes cache utilization and reduces memory access time. This merging also contributes to parallel processing, allowing cohesive data sets to be operated on in parallel. Generally speaking, in these embodiments, the transformation to a single OS enhances computational efficiency, reduces memory overhead, and promotes a more cohesive and accessible representation of underlying space-time information.

[0089] In some embodiments, the transformation process may be performed by Figure 7 The temporary OS in the algorithm replaces the TOS in luvHarris; but it is different from the per-event (2k+1) in luvHarris. 2 Compared to writing 1 bit per event, OS only needs D writes. Although OS has both spatial and temporal information, the temporal information is based on a global ordering and therefore spans a much larger range of values than the 8 bits used in TOS. This is corrected in the next step of tile creation by permutation normalization.

[0090] In these embodiments, the OS only needs to write D per event, which requires much less write time than luvHarris. Although the method according to these embodiments utilizes a logical pointer (i.e., a row pointer or a column pointer) that represents the current entry in the process, the movement of the logical pointer across the physical rows of the memory is facilitated by a loop operation determined by the modulus (mod) operation in the above steps, thereby ensuring that the logical pointer cycles through the rows of the memory in a circular manner.

[0091] In another embodiment, the OS matrix is derived from some of the adjacent rows of the recorded events in the 2D array. For example, the OS matrix is derived from three adjacent rows in the 2D array, and multiple OS matrices are transformed from one 2D array.

[0092] In this embodiment, the method or system incorporating this transformation is optimized for a DVS with a higher sensitivity or sensing rate.

[0093] In yet another embodiment, the OS matrix is derived from one of the rows of the recorded events in the 2D array. Thus, multiple OS matrices are transformed from one 2D array, and each OS matrix corresponds to the most recent row of the recorded events.

[0094] In this embodiment, a method or system incorporating this transformation is optimized for DVS with high throughput per second and provides adaptive corner detection with high sensitivity.

[0095] The following description refers to Figure 8 . In another embodiment, the method includes extracting an image patch S34 of elements in the ordered surface matrix; applying a Harris detector to the image patch to generate a Harris score S35; and comparing the Harris score generated by the Harris detector with a predefined threshold S36.

[0096] In this embodiment, an image patch surrounding the event of interest is extracted from the OS for use in the Harris detector or Harris calculation in order to utilize the sparsity of the events. Steps S34, S35, S36 optimize the process for efficient feature detection. The steps define a systematic method for selecting a region or image patch centered on a specific event from the OS, thus allowing the Harris detector to focus on the specific spatial location of interest within the OS.

[0097] A notable feature of this image patch extraction process is its parameter tunability, such that the algorithm can dynamically adjust the size and configuration of the image patch. This flexibility ensures that the Harris detector can effectively capture and analyze features at different scales, contributing to the robustness of the algorithm in identifying corners and related patterns in visual data.

[0098] Additionally, step S34 typically incorporates considerations for adjacent events and their contribution to the image patch. This ensures that the Harris detector exploits the spatio-temporal correlations present in the OS matrix, enhancing its ability to discriminate meaningful features. The adaptive and parameterizable nature of the image patch extraction process plays a key role in the overall success of the Harris detector, making it well-suited for various applications in data processing and computer vision.

[0099] In various embodiments, the image patch has M rows and N columns, and M ranges from 5 to 11, and N ranges from 5 to 11. Thus, the method or system of the embodiments can capture significant information stored in the OS while consuming minimal memory when using the Harris detector. Additionally, the image patch size plays a crucial role in corner detection such as using the Harris detector, especially for images with resolutions such as 240x180 or 360x260. In certain embodiments, it is beneficial to select an alternative image patch size based on the varying resolution of the event camera. In other words, the methods and systems in these embodiments provide flexibility in selecting the image patch size. This adaptability allows the methods and systems to be used with various types of event cameras, thus ensuring effective corner detection across different scenarios.

[0100] In some embodiments, the values of M and N can be the same, and the method utilizes square image patches. Thus, the output of the Harris detector can be determined by referring to existing data sets to improve efficiency. For example, both M and N can be 5, 7, 9, 11, or any suitable odd number, depending on the resolution of the event camera. Thus, the method according to the embodiments of the present invention is flexible and applicable to systems with different event camera resolutions.

[0101] In some embodiments, permutation normalization can be applied to the OS matrix before performing corner detection. The following description refers to Figure 9 . The method includes performing permutation normalization S45 on the image patch of elements before employing the Harris detector.

[0102] The permutation normalization of these embodiments modifies the range of the values of the time information in the OS matrix. Specifically, the steps bypass a method involving expensive division operations. In this procedure S45, the non - zero values within the image patch are collected and arranged in descending order within a queue. Subsequently, these values undergo permutation normalization, where the normalized pixel value is calculated as 255 minus the index. Here, the index represents the position of the pixel within the permuted queue. This method not only optimizes the computational efficiency but also conforms to theoretical concepts such as rank - order coding, reflecting a sophisticated strategy reminiscent of visual cortical coding. The result is a simplified and adaptable normalization method that significantly enhances the overall performance of the system or method in visual data processing applications.

[0103] In some embodiments, local image patch construction and Harris calculation can be performed by, for example, Figure 10The algorithm 2 shown represents that the global sorting information in the OS spans a wide range of values (0 - CXD), which makes it difficult to perform Harris evaluation. Normalizing the global index to span the range of image patch indices would generally require expensive division operations. In practice, the non-zero values in the image patch are added to a queue and sorted in descending order. Then, based on the order, the values are normalized such that the normalized pixel value is equal to 255 - index, where index refers to the position of the pixel in the sorted queue - we refer to this process as sorted normalization. Once the image patch is extracted and normalized, we can apply the Harris detector to the image patch, generate a score for the event of interest, and classify it as a corner if the score exceeds a threshold. As previously mentioned, we expect to use in-memory computing (IMC) methods to accelerate Harris evaluation. Additionally, we perform this calculation on all events stored in the most recent row within a batch of C - this provides more context to correctly classify corner events.

[0104] The methods and systems of embodiments of the present invention provide event-based corner detection for edge devices that uses minimized memory. Despite the small memory footprint and energy usage, the methods and systems achieve accuracy comparable to / better than luvHarris.

[0105] In various embodiments, in-memory computing (IMC) techniques are utilized to accelerate processing, and adaptive corner detection can be performed by a machine vision processor implemented using a specially configured computing device, computer processor, or electronic circuitry system, the specially configured computing device, computer processor, or electronic circuitry system including but not limited to application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), microcontrollers, and other programmable logic devices configured or programmed according to the teachings of the present disclosure. Machine instructions running in the computing device, computer processor, or programmable logic device can be readily made by practitioners in the software or electronics arts based on the teachings of the present disclosure.

[0106] All or part of the method according to an embodiment can be executed in one or more computing devices including a server computer, a personal computer, a laptop computer, a mobile computing device (such as a smartphone), and a tablet computer.

[0107] Embodiments may include a computer storage medium, a transient and non-transient memory device having machine instructions stored therein, the machine instructions being usable to program or configure a computing device, a computer processor, or an electronic circuitry system to perform any of the processes of the present invention. The storage medium, transient and non-transient memory device may include (but is not limited to) a floppy disk, an optical disk, a Blu-ray disc, a DVD, a CD-ROM, and a magneto-optical disk, a ROM, a RAM, a flash memory device, or any type of medium or device suitable for storing instructions, code, and / or data.

[0108] Each of the functional units and modules according to various embodiments may also be implemented in a distributed computing environment and / or a cloud computing environment, in which all or part of the machine instructions are executed in a distributed manner by one or more processing devices interconnected by a communication network such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0109] For purposes of illustration and description, the foregoing description of the invention has been provided. It is not intended to be exhaustive or to limit the invention to the precise form disclosed. Many modifications and variations will be apparent to practitioners in the art.

[0110] The embodiments were chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention in various embodiments and various modifications suitable for the particular use contemplated.

Claims

1. A method for adaptive corner detection in machine vision, characterized in that, Comprising: Obtaining one or more event data from a dynamic vision sensor; Capturing, by a machine vision processor, a plurality of recorded events of the event data and organizing them into a 2D array; Transforming, by the machine vision processor, the 2D array into one or more ordered surface matrices by filling one or more empty image matrices with the recorded events based on the coordinates of the recorded events and assigning order values; Applying, by the machine vision processor, a corner detector to the ordered surface matrix; And Outputting a corner detection result from the corner detector.

2. The method according to claim 1, wherein The recorded events are arranged in a globally time-ordered layout in the 2D array.

3. The method according to claim 1, wherein The recorded events are arranged in a plurality of rows in the 2D array, and each of the rows has a series of the recorded events arranged in a globally time-ordered layout; and The number of the recorded events in the rows is the same, and the rows are arranged in a globally time-ordered layout.

4. The method according to claim 3, wherein The range of the maximum number of the recorded events in each row is from 25 to 100.

5. The method according to claim 3, characterized in that, Each ordered surface matrix is derived from all of the rows of the recorded events in the 2D array.

6. The method according to claim 1, wherein The step of applying the corner detector to one of the ordered surface matrices includes: Extracting image patches of elements in the ordered surface matrix; Applying a Harris detector to the image patches to generate Harris scores; And Comparing the Harris scores generated by the Harris detector with a predefined threshold.

7. The method according to claim 6, characterized in that, Further comprising: Performing permutation normalization on the image patches of the elements before applying the Harris detector.

8. The method according to claim 6, characterized in that, The image patches have M rows and N columns, and the range of M is from 7 to 11, and the range of N is from 7 to 11.

9. The method according to claim 1, characterized in that Further comprising: Applying a spatio-temporal correlation filter to the event data before organizing the recorded events into the 2D array.

10. A system for adaptive corner detection in machine vision, characterized in that, Comprising: A dynamic vision sensor; and A machine vision processor electrically connected to the dynamic vision sensor; Wherein the machine vision processor is configured to: Obtain one or more event data from the dynamic vision sensor; [[ID= ​ ​ ​ 11. The system according to claim 10, wherein ​ 12. The system according to claim 10, characterized in that, ​ 13. The system according to claim 12, wherein, ​ 14. The system according to claim 12, wherein ​ 15. The system according to claim 10, wherein ​ Extract image patches of elements in the ordered surface matrix; The machine vision processor applies a Harris detector to the image patches to generate Harris scores; And The machine vision processor compares the Harris scores generated by the Harris detector with a predefined threshold.

16. The system according to claim 15, wherein The machine vision processor is further configured to perform permutation normalization on the image patches of elements before applying the Harris detector.

17. The system according to claim 16, wherein, The image patches have M rows and N columns, and the range of M is from 7 to 11, and the range of N is from 7 to 11.

18. The system according to claim 17, wherein The machine vision processor is further configured to apply a spatio-temporal correlation filter to the event data before organizing the recorded events into the 2D array.

Images