System and method for determining object coordinates in an environment

Abstract

A method (100) for determining object coordinates in an environment is disclosed. The method (100) comprises: receiving asynchronous event data for the environment from an event-based vision sensor (300); generating a spatio-temporal tensor representation from the received asynchronous event data by sampling a predetermined number of latest non-zero positions in a time dimension at each spatial position; generating temporal features by processing the spatio-temporal tensor representation by means of a forked temporal processing module; generating spatio-temporal features from the temporal features using a recurrent transformer comprising a plurality of sequential transformer blocks; determining object positions in pixel coordinates based on the generated spatio-temporal features; and determining coordinate information of an object by converting the determined object positions from the pixel coordinates to world coordinates.

Description

Technical Field

[0001] This invention relates generally to object detection systems and methods, and more specifically to a method and system for determining object coordinates in an environment using an event-based visual sensor and a tensor-based event representation for high-speed object detection applications. Background Technology

[0002] Event-based vision sensors represent a fundamental advancement in imaging technology, operating differently from traditional frame-based image acquisition systems. These sensors generate asynchronous event data in response to localized pixel-level changes in light intensity, producing discrete temporal events rather than synchronous frame sequences. Each event includes specific data parameters, such as timestamp information, pixel coordinates, polarity values, and resolution parameters, enabling high-speed visual data acquisition with microsecond-level temporal accuracy.

[0003] In applications requiring real-time object coordinate determination, especially in dynamic environments, traditional imaging systems face significant technical limitations. For example, the ability to detect and track objects (especially pedestrians) in real-time poses a considerable challenge in autonomous vehicle applications and pedestrian safety systems. Traditional frame-based cameras operating at standard frame rates introduce substantial processing latency, typically 33 milliseconds per frame at 30 frames per second. This inherent latency becomes particularly problematic in scenarios requiring rapid object detection and coordinate determination, such as pedestrian safety systems and autonomous navigation applications, where immediate response time is critical to system effectiveness.

[0004] Current technological solutions employ a variety of sensing modalities, including radar systems, optical detection and ranging (LiDAR) devices, and traditional frame-based cameras. These systems typically process environmental data using standard computer vision algorithms and neural network architectures designed to synchronize data streams. Traditional methods often implement basic temporal integration techniques to process sensor data and determine object coordinates using standard perspective transformation techniques.

[0005] However, existing solutions exhibit significant technical limitations. Frame-based imaging systems suffer from limited dynamic range and motion blur during high-speed object movement. While lidar and radar systems provide direct distance measurements, they generate spatially sparse data points and operate within a fixed scan frequency. Furthermore, current object detection implementations primarily utilize simple continuous event representations, which fail to effectively leverage the asynchronous nature and high temporal resolution of event-based sensor data, resulting in suboptimal accuracy in temporal feature extraction and coordinate determination.

[0006] Therefore, there is a need for improved object detection systems that can effectively utilize the high temporal resolution and asynchronous nature of event-based visual sensors while maintaining computational efficiency and detection accuracy in dynamic environments. Summary of the Invention

[0007] This invention aims to provide a method and system for determining the coordinates of objects in an environment using an event-based visual sensor. The method combines tensor-based event representation with advanced spatiotemporal processing techniques to achieve high-speed object detection with reduced latency.

[0008] In one aspect, the present invention provides a method for determining object coordinates, the method comprising receiving asynchronous event data from an event-based vision sensor and generating a spatiotemporal tensor representation by sampling a predetermined number of latest non-zero positions in the time dimension at each spatial location. The method further processes the representation by a bifurcated temporal processing module and a cyclic transformer comprising multiple sequential transformer blocks to generate spatiotemporal features for object detection.

[0009] This spatiotemporal tensor representation preserves both spatial and temporal information, while maintaining polarity values ​​and enabling asynchronous processing based on event triggering frequency. The representation includes event timestamp information, event pixel coordinates, event polarity data, and resolution parameters.

[0010] On the other hand, the present invention implements a new method for time feature processing through a bifurcation time processing module, which directly provides time features to the cyclic transformer without implementing a multilayer perceptron layer, thereby improving processing efficiency while maintaining feature quality.

[0011] The system achieves coordinate determination through a multi-stage process involving perspective transformation to convert pixel coordinates to world coordinates, linear motion modeling for trajectory estimation, and coordinate refinement based on the estimated trajectory. Attached Figure Description

[0012] The invention and its many accompanying aspects can be more fully understood by referring to the following description taken in conjunction with the accompanying drawings:

[0013] Figure 1 This is a flowchart of a method for determining the coordinates of an object in an environment according to one or more embodiments of the present invention;

[0014] Figure 2 This is a block diagram representation of a system for determining the coordinates of objects in an environment according to one or more embodiments of the present invention;

[0015] Figure 3 A typical setup for an event camera implementation according to one or more embodiments of the present invention is shown;

[0016] Figure 4 This is a block diagram illustrating a processing pipeline for an object detection algorithm according to one or more embodiments of the present invention; and

[0017] Figure 5 Exemplary outputs from an object detection algorithm, illustrating object detection and tracking results according to one or more embodiments of the present invention, are depicted. Detailed Implementation

[0018] Various embodiments are described with reference to the accompanying drawings, wherein similar reference numerals are used throughout to refer to similar elements. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of one or more embodiments. It will be apparent that such embodiments may be practiced without these specific details.

[0019] This document discloses examples of methods, systems, computer programs, and computer-readable media for determining the coordinates of objects in an environment. In the following description, numerous specific details are set forth for purposes of explanation in order to provide a thorough understanding of embodiments of the invention. However, it will be apparent to those skilled in the art that embodiments of the invention may be practiced without these specific details or with equivalent arrangements. In other instances, well-known structures and apparatuses are shown in block diagram form to avoid unnecessarily obscuring embodiments of the invention.

[0020] refer to Figure 1 A flowchart of a method for determining object coordinates in an environment (as indicated by reference numeral 100) according to an embodiment of the present invention is shown. The method 100 of this disclosure is implemented using an event-based vision sensor. As used herein, "object coordinates" includes spatial location data of a detected object relative to an event-based vision sensor, wherein the object may include pedestrians, vehicles, or other moving entities within the field of view of the event-based vision sensor. The environment includes the physical space within the detection range of the event-based vision sensor, typically extending 30 meters from the sensor location. The environment may include roadways, sidewalks, intersections, and surrounding infrastructure.

[0021] Method 100 for determining object coordinates in an environment processes asynchronous event data received from an event-based vision sensor to generate a spatiotemporal tensor representation. This spatiotemporal tensor representation includes a structured data array containing event timestamp information, event pixel coordinates, event polarity data, and event resolution parameters. The method processes the spatiotemporal tensor representation through multiple sequential processing stages to generate spatiotemporal features, determine object positions in pixel coordinates, and convert the determined object positions into world coordinates. The processing stages implement specific data transformation operations to maintain the temporal order of events while simultaneously enabling efficient parallel processing of spatial information for coordinate determination.

[0022] refer to Figure 2 This diagram illustrates a block diagram of a system 200 for determining the coordinates of objects in an environment according to one or more embodiments of the present invention. It is understood that the system 200 described herein can be implemented in various forms of hardware, software, firmware, dedicated processors, or combinations thereof. One or more embodiments in this embodiment may take the form of a computer program product including program modules accessible from a computer-usable or computer-readable medium, which stores program code for use by or with one or more computers, processors, or instruction execution systems. For the purposes of this description, a computer-usable or computer-readable medium can be any device that can contain, store, transmit, propagate, or transfer a program for use by or with an instruction execution system, device, or apparatus. The medium can be an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system (or device or apparatus), or a medium that is itself a signal carrier or a propagation medium within a signal carrier, not included in the definition of a physical computer-readable medium, including semiconductor or solid-state memory, magnetic tape, removable computer floppy disks, random access memory (RAM), read-only memory (ROM), rigid disks, and optical disks, such as optical disc read-only memory (CD-ROM), optical disc readers, and digital versatile optical discs (DVDs). The processors and program code used to implement each aspect of the technology can both be centralized or distributed (or a combination thereof), as is known to those skilled in the art.

[0023] In the example, system 200 may be embodied as a computer program programmed to determine the coordinates of objects in an environment. System 200 may be incorporated into one or more physical packages (e.g., chips). For example, a physical package includes an arrangement of one or more materials, components, and / or wires on a structural assembly (e.g., a substrate) to provide one or more properties, such as physical strength, dimensional retention, and / or limitation of electrical interactions. It is envisioned that in some embodiments, a computing device may be implemented in a single chip. As shown, system 200 includes communication mechanisms, such as a bus 202 for transferring information between components of system 200. System 200 includes one or more processing units 204 and one or more memory units 206. Hereinafter, memory units 206 are communicatively coupled to processing units 204. In the example, memory unit 206 may be embodied as a computer-readable medium on which program code segments of a computer program are stored, which can be loaded into and / or executed in the system to cause system 200 to perform steps for performing the stated purpose.

[0024] Generally, as used herein, the term "processing unit" refers to a computing element operable to respond to and process instructions from the drive system 200. Optionally, a processing unit includes, but is not limited to, a microprocessor, microcontroller, Complex Instruction Set Computing (CISC) microprocessor, Reduced Instruction Set Computing (RISC) microprocessor, Very Long Instruction Word (VLIW) microprocessor, or any other type of processing circuitry. Furthermore, the term "processing unit" can refer to one or more individual processors, processing devices, and various elements associated with the processing device that can be shared by other processing devices. Additionally, the one or more individual processors, processing devices, and elements are arranged in various architectures to respond to and process instructions from the drive system 200.

[0025] In this document, memory unit 206 may be volatile memory and / or non-volatile memory. Memory unit 206 may be coupled for communication with processing unit 204. Processing unit 204 may execute instructions and / or code stored in memory unit 206. Various computer-readable storage media may be stored in and accessed from memory unit 206. Memory unit 206 may include any suitable elements for storing data and machine-readable instructions, such as read-only memory, random access memory, erasable programmable read-only memory, electrically erasable programmable read-only memory, hard disk drive, removable media drive for disposing of optical discs, digital video discs, floppy disks, magnetic tape cassettes, memory cards, etc.

[0026] Specifically, processing unit 204 has connectivity to bus 202 to execute instructions and process information stored in memory unit 206. Processing unit 204 may include one or more processing cores, each configured to execute independently. Multi-core processors enable multiprocessing within a single physical package. Examples of multi-core processors include two, four, eight, or more processing cores. Alternatively or additionally, processing unit 204 may include one or more microprocessors configured sequentially / in series via bus 202 to enable independent execution, pipelining, and multithreading of instructions. Processing unit 204 may also be accompanied by one or more specialized components, such as one or more digital signal processors (DSPs) and / or one or more application-specific integrated circuits (ASICs), for performing certain processing functions and tasks. Other specialized components used to help perform the innovative functions described herein include one or more field-programmable gate arrays (FPGAs) (not shown), one or more controllers (not shown), or one or more other special-purpose computer chips.

[0027] System 200 may further include interface 208, such as a communication interface (the terms are used interchangeably herein), which enables system 200 to communicate with other systems to receive and send information. Communication interface 208 may include a medium (e.g., a communication channel) through which system 200 communicates with other systems. Examples of communication interface 208 include, but are not limited to, communication channels in a computer cluster, local area communication channels (LANs), cellular communication channels, wireless sensor communication channels (WSNs), cloud communication channels, metropolitan area communication channels (MANs), and / or the Internet. Optionally, communication interface 208 may include one or more of wired connections, wireless networks, cellular networks (such as 2G, 3G, 4G, 5G mobile networks), and Zigbee connections.

[0028] System 200 also includes database 210. As used herein, database 210 is an organized collection of structured data, typically stored in a computer system and designed to be easily accessed, managed, and updated. Database 210 may take the form of a central information repository that can be queried, analyzed, and processed to support various applications and business processes. In system 200, database 210 provides mechanisms for storing, retrieving, updating, and deleting data, and typically includes features such as data validation, security, backup and recovery, and data modeling.

[0029] System 200 further includes an input device 212 and an output device 214. The input device 212 can take various forms, depending on the specific application of system 200. In this example, the input device 212 may include one or more of the following: a keyboard, a mouse, a touchscreen display, a microphone, a camera, or any other hardware component that enables a user to interact with system 200. Furthermore, the output device 214 may take the form of a display, a printer, a communication channel, etc., without any limitation.

[0030] In the current system 200, the processing unit 204 and accompanying components have connectivity to the memory unit 206 via bus 202. The memory unit 206 includes both dynamic memory (e.g., RAM, disk, writable optical disc, etc.) and static memory (e.g., ROM, CD-ROM, etc.) for storing executable instructions that, when executed, perform the method steps described herein for determining object coordinates in an environment. Specifically, the memory unit 206 includes an object coordinate determination module 216 for performing the steps for said purpose.

[0031] Combination Reference Figures 1 to 2 The document describes the various steps of a method 100 (described below) for determining the coordinates of objects in an environment, which may be performed in system 200 or more specifically in processing unit 204 of system 200. It is understood that although method 100 is shown and described as a sequence of steps, it is contemplated that various embodiments of method 100 may be performed in any order or different combinations, and need not include all the steps shown.

[0032] At step 102, method 100 includes receiving asynchronous event data of the environment from an event-based vision sensor. (See reference...) Figure 3 The figure illustrates a configuration for implementing an event-based vision sensor (as indicated by reference numeral 300). As shown, the event-based vision sensor 300 is mounted on a vehicle to receive asynchronous event data from the environment. The event-based vision sensor 300 may include an independent array of pixels that respond to changes in logarithmic light intensity. The mounting configuration of the event-based vision sensor 300 provides a field of view (FOV) extending outwards from the front of the vehicle. The FOV covers the forward-looking area used for detecting pedestrians and other objects in the environment.

[0033] The event-based vision sensor 300 operates on the principle of asynchronous event generation, where each pixel independently responds to logarithmic changes in light intensity. The sensor generates an event when the logarithmic change in intensity at a pixel location exceeds a predetermined threshold (typically ±15% of the previous intensity value). Each event e is characterized by a tuple (x, y, t, p), where (x, y) represents the pixel coordinates, t represents a timestamp with microsecond-level precision, and p indicates the polarity (±1) corresponding to the intensity changing in an increasing or decreasing manner.

[0034] Asynchronous event data received from the event-based vision sensor 300 maintains temporal accuracy without being constrained by a fixed frame rate. This enables continuous monitoring of the environment with minimal latency between object motion and event generation. Independent operation of each pixel in the event-based vision sensor 300 allows for efficient data generation, specifically focused on areas of the scene where intensity changes occur.

[0035] At step 104, method 100 includes generating a spatiotemporal tensor representation from the received asynchronous event data by sampling a predetermined number of latest non-zero positions in the time dimension at each spatial location. Now refer to... Figure 4 This paper presents a tensor-based event processing architecture 400. Asynchronous event data (represented as "events") is received and processed in multiple stages to generate a spatiotemporal tensor representation. The spatiotemporal tensor representation implements a temporal active focusing (TAF) sampling scheme, which processes events at time intervals (tk-1,k) and (tk-2,tk-1) to capture temporal dynamics.

[0036] In this embodiment, generating the spatiotemporal tensor representation involves generating a four-dimensional tensor from the received asynchronous event data, including spatial coordinates, temporal information, and polarity values. Specifically, the TAF sampling operation generates a four-dimensional tensor T ∈ ℝ^(H×W×K×2), where H and W correspond to the spatial dimensions of the sensor array, and K represents a predetermined number of temporal positions (K=8 in the illustrated embodiment). In this paper, the last dimension (2) accommodates positive and negative polarity values.

[0037] The spatiotemporal tensor represents the temporal ordering of events within each spatial coordinate location (x, y). Further, method 100 implements an adaptive time window at each spatial coordinate location based on event frequency. In the given example, the TAF tensor (K=8) samples the eight nearest non-zero locations in the temporal dimension at each spatial location, thereby implementing an adaptive time window adjusted based on local event frequency. The sampling operation preserves both spatial relationships and temporal sequence information from the asynchronous event stream.

[0038] Furthermore, such as Figure 4As shown, TAF represents the concatenation of multiple time slices of event data. Each time slice includes events that occur within specific time intervals (tk-1, k) and (tk-2, tk-1), where k represents the current time step. Time slices are processed through parallel paths that maintain temporal ordering while enabling efficient parallel processing of spatial information.

[0039] The spatiotemporal tensor representation contains multiple data components structured to enable efficient event processing. In this paper, the spatiotemporal tensor representation preserves spatial and temporal information from asynchronous event data. This preservation is achieved through a four-dimensional tensor structure, where the spatial dimension corresponds to the sensor array resolution, K=8 time sampling positions maintain time-series information, and the final dimension preserves binary polarity values. The spatiotemporal tensor representation also maintains polarity values ​​through dedicated channels within the tensor structure. Furthermore, the spatiotemporal tensor representation implements asynchronous processing based on event triggering frequency, achieved through an adaptive time window.

[0040] According to embodiments of this disclosure, the spatiotemporal tensor representation integrates event timestamp information, event pixel coordinates, event polarity data, and event resolution parameters into a unified data structure. In this document, the event timestamp information includes a timestamp (with a resolution of approximately 1 μs) indicating when each pixel exceeds an intensity change threshold. The timestamp values ​​maintain a strict temporal order within each spatial location. The event pixel coordinates represent the spatial location of each event within the sensor array. The event polarity data p indicates the direction of intensity change for each event. The event resolution parameters include the spatial dimension of the tensor structure and the temporal sampling rate. This structured representation enables subsequent processing stages to extract both spatial and temporal features while maintaining the asynchronous nature of the original event data.

[0041] At step 106, method 100 includes generating temporal features by processing the spatiotemporal tensor representation using a bifurcation time processing module. For example... Figure 4 As shown, the Bifurcation Time Processing (BFM) module processes the spatiotemporal tensor representation through a dedicated processing pipeline to generate temporal features. BFM receives a four-dimensional tensor T(x,y,t,p) from the TAF representation and implements a bifurcation processing structure that operates in the temporal dimension while maintaining spatial coherence.

[0042] In this paper, the processing via the bifurcation time processing module includes processing the time dimension data through a fully connected layer. This is performed within BFM, which involves operations at K time locations at each spatial location (x, y). The fully connected layer processes the time dimension values, thereby establishing weighted connections between time locations. The fully connected layer maintains polarity information while transforming the time data according to the mapping function f(T) = W·T + b, where W represents the weight matrix and b represents the bias term. Method 100 further includes progressively aggregating the processed time dimension data at adjacent time locations. This step implements a hierarchical combination operation. For each spatial location (x, y), BFM aggregates time data from adjacent locations based on the temporal distance between events. The aggregation operation uses a weighted average operation to combine the temporal features from locations k and k+1. Method 100 further includes assigning weights to the aggregated data based on the temporal recentity of the aggregated data. This step implements an exponential decay function applied to the time locations. For each time location k, the weights are calculated. The weighting scheme used assigns higher importance to more recent time information while maintaining a continuous representation of the temporal evolution of events.

[0043] At step 108, method 100 includes using a cyclic transformer comprising multiple sequential transformer blocks to generate spatiotemporal features from temporal features. For example... Figure 4 As shown, the spatiotemporal feature generation implementation comprises a loop transformer consisting of three sequential RVT (Recurrent Visual Transformer) blocks. Each RVT block processes temporal features through a structured pipeline that includes local feature interaction, global feature blending, and temporal feature extraction stages. The RVT blocks operate sequentially, with each block taking the output features from the previous block as input.

[0044] In this paper, generating spatiotemporal features involves using a convolutional neural network (CNN) to process temporal features, arranging pixels in a two-dimensional grid within a computational graph. The CNN processing stage arranges pixels in the two-dimensional grid using Conv2D layers. The Conv2D layers process temporal features with a 3×3 kernel size and a stride of 2. The computational graph arrangement enables parallel processing of spatial relationships while maintaining the temporal coherence established by BFM. The Conv2D layers generate feature maps with different spatial resolutions through progressive downsampling operations. Method 100 further includes transforming the arranged pixels using a multi-axis self-attention layer to generate transformed features. The pixel arrangement via the multi-axis self-attention layer transformation achieves... Figure 4 The block self-attention mechanism is shown in the RVT block architecture. Method 100 further includes applying a self-attention operation to the transformed features to extract global spatial features. A grid self-attention operation is applied to the transformed features to extract global spatial features via an attention mechanism. Global feature extraction operates across the entire feature map to capture long-range spatial dependencies. Figure 4 As shown, the LSTM components process these global features sequentially, thereby maintaining temporal state information across multiple time steps. The LSTM state from the previous time step is used to compute features for the next processing stage, where the results are saved for use in subsequent time steps in the pipeline.

[0045] As used in this paper, a Convolutional Neural Network (CNN) is a multi-layered feedforward neural network made by stacking many hidden layers sequentially on top of each other. This sequential design allows CNNs to learn hierarchical features. Hidden layers are typically convolutional layers, followed by activation layers, some of which are then pooling layers. CNNs can be configured to recognize patterns in data. Convolutional layers can include convolutional kernels used to find patterns across the input data. A convolutional kernel returns a large positive value for a portion of the input data that matches the kernel's pattern and a smaller value for another portion that does not match the kernel's pattern. CNNs can extract informative features from training data without requiring manual processing of that training data. CNNs can produce accurate results in scenarios involving large amounts of unstructured data, such as image classification, speech recognition, and natural language processing. Furthermore, CNNs are computationally efficient because they can assemble patterns of increasing complexity using relatively small kernels in each hidden layer.

[0046] According to embodiments of this disclosure, the cyclic converter processes time characteristics through three sequential converter blocks. For example... Figure 4 As shown, the cyclic transformer implements a three-block sequential processing architecture, where each transformer block performs a specialized attention operation. This sequential arrangement progressively refines spatiotemporal features through hierarchical processing stages. In this paper, the first transformer block implements spatial self-attention processing on temporal features. The first transformer block receives temporal features from the bifurcated temporal processing module and implements spatial self-attention processing to compute attention weights. This operation captures spatial relationships while maintaining the temporal structure of the input features. The second transformer block implements combined spatial and temporal attention processing on the output from the first transformer block. The combined attention mechanism handles both spatial and temporal dimensions simultaneously through a unified attention operation. This block integrates spatial relationships and temporal dependencies. The third transformer block implements refined temporal attention processing on the output from the second transformer block to generate spatiotemporal features. This block generates spatiotemporal features through focused temporal attention operations. The output features incorporate refined temporal dependencies while maintaining spatial coherence / consistency.

[0047] In this embodiment, the bifurcation time processing module processes the spatiotemporal tensor representation by directly providing the time features to the cyclic transformer, without implementing a multilayer perceptron layer. In other words, BFM implements a direct-connect architecture that provides time features to the cyclic transformer without intermediate multilayer perceptron layers. Figure 4 As shown, the BFM output is directly connected to the first RVT block. This architectural modification reduces computational complexity while maintaining feature quality through subsequent transformer processing stages. Removing the multilayer perceptron layer eliminates redundant feature transformation operations that are efficiently handled by the transformer block.

[0048] At step 110, method 100 includes determining the object's location in pixel coordinates based on the generated spatiotemporal features. (See reference) Figure 4 Object locations in pixel coordinates are determined by processing spatiotemporal features through an Agile Event Detector (AED) architecture. The AED implements a Feature Pyramid Network (FPN) structure, which operates on the spatiotemporal features generated by a third RVT block to produce object detection outputs at multiple spatial scales. Figure 5 An exemplary detection output (as indicated by reference numeral 500) from an event-based detection system is shown, which performs object location determination in pixel coordinates. The detection output displays event data as white dots on a black background, where each dot represents a single event generated by the event-based vision sensor. The detection system performs bounding box localization with confidence scores for multiple object categories. Pedestrian detection is shown with a bounding box and an associated confidence score of 0.8, indicating high detection confidence. Additional objects are detected with a confidence score of 0.9, demonstrating multi-object detection capability. The event points scattered around the detected objects represent asynchronous events generated by the event-based vision sensor in response to changes in scene intensity. The pixel coordinates of the bounding boxes provide the spatial localization of the detected objects within the field of view of the event-based vision sensor 300, enabling subsequent coordinate transformation to world coordinates for distance and trajectory estimation.

[0049] In this embodiment, determining the object location includes processing spatiotemporal features generated by an event detector configured for high inference speed to generate initial object detection data. Processing the spatiotemporal features by the event detector includes a detection head that operates on feature maps at multiple scales. The AED processes the generated spatiotemporal features through a high-speed inference pipeline optimized for asynchronous event data processing. For each spatial location, the detection head outputs a classification score for the object category, bounding box coordinates relative to predefined anchor points, and an object confidence score. Method 100 further includes refining the initial object detection data based on the temporal resolution of the asynchronous event data. The refinement of the initial object detection data is based on adaptive temporal processing using temporal information (microsecond resolution) from the asynchronous event data. The detection refinement process maintains a separate detection path, such as... Figure 4As shown, the parallel AED module processes the time series of spatiotemporal features to generate and refine object detection. The refinement process includes temporal consistency checks across multiple detection windows, each corresponding to a time slice of event data. The temporal resolution of the asynchronous event data enables accurate tracking of object motion across detection windows, allowing for refinement of object location based on the temporal coherence of the detected features.

[0050] In some embodiments, method 100 further includes performing event-based data augmentation. This involves performing time jitter on asynchronous event data while maintaining temporal causality, performing polarity reversal while maintaining spatiotemporal consistency, and performing spatial warping via a perspective transformation matrix while maintaining event temporal order. The AED architecture incorporates data augmentation methods that include performing time jitter on asynchronous event data while maintaining temporal causality, performing polarity reversal while maintaining spatiotemporal consistency, and performing spatial warping via a perspective transformation matrix while maintaining event temporal order. Such operations can be conceived by those skilled in the art and are therefore not further described herein. These augmentation techniques enhance the robustness of object location determination while preserving the inherent temporal precision in asynchronous event data.

[0051] At step 112, method 100 includes determining the object's coordinate information by converting the determined object position from pixel coordinates to world coordinates. (Reference) Figure 4 The determination of coordinate information enables a coordinate transformation pipeline for converting pixel coordinates to world coordinates. The coordinate transformation process includes a perspective transformation matrix and linear motion modeling for trajectory estimation. The transformation from pixel coordinates to world coordinates utilizes camera calibration parameters, including the intrinsic camera matrix K, rotation matrix R, and translation vector t. The perspective transformation matrix P achieves the transformation from pixel coordinates (u,v) to world coordinates (X,Y,Z) by applying a geometric transformation that considers the camera's position and orientation relative to the detected object.

[0052] In this paper, determining coordinate information includes estimating the distance to the detected object based on the temporal changes in the determined object's position. The object distance estimation utilizes the temporal changes in the detected object's position by analyzing bounding box measurements across sequential detection windows. The distance estimation process implements a linear motion model based on the temporal changes in the object's size and position within the field of view. The distance D to the detected object is calculated based on the relationship between the focal length f, the assumed real-world object height H_real, and the measured object height H_pixel in pixels. Temporal consistency of the distance estimation results is maintained by integrating multiple measurements across sequential detection windows. Method 100 further includes determining the world coordinates of the object based on the estimated distance. This process is performed in conjunction with pixel coordinate information to generate 3D position data in a world reference frame. World coordinate calculation uses calibrated / standardized camera parameters to apply perspective transformations to the pixel coordinates. It can be understood that the coordinate determination process maintains temporal coherence by sequentially updating the position estimation results based on new detection data.

[0053] In this paper, determining coordinate information further includes implementing a perspective transformation to convert pixel coordinates into initial world coordinates. The perspective transformation utilizes a calibrated transformation matrix to convert pixel coordinates (u,v) into initial world coordinates (X,Y,Z), where K represents the camera intrinsics, R represents the rotation matrix, and t indicates a translation vector defining the camera position relative to the world reference frame. Method 100 further includes applying linear motion modeling to the initial world coordinates to estimate the trajectory. The linear motion modeling stage processes the initial world coordinates through a state estimation framework that includes position and velocity components. The state vector represents the object's position and velocity in world coordinates, thereby achieving trajectory estimation through sequential state updates. Method 100 further includes refining the world coordinates based on the estimated trajectory. The refinement of the world coordinates utilizes the estimated motion parameters to achieve trajectory-based optimization. The refinement process integrates multiple coordinate measurements across a time window to enhance position accuracy through statistical filtering of the coordinate estimation results.

[0054] The complete coordinate determination process transforms the detected object's position from image space coordinates to physical space coordinates, facilitating subsequent tracking and trajectory estimation operations. World coordinate representation provides measurement results suitable for preventative actions based on the detected object's position and motion within the environment. The coordinate transformation pipeline and... Figure 4 The object detection framework shown is integrated to provide location information for determining preventative actions using transformed coordinates. This integration enables real-time tracking and response capabilities based on accurate world coordinate measurements of the detected objects.

[0055] This method 100, used to determine object coordinates in an environment using event-based visual sensors, achieves significant technological advancements in object detection and coordinate determination. Method 100 introduces a spatiotemporal tensor representation that preserves both spatial and temporal information from asynchronous event data, while maintaining polarity values ​​and implementing asynchronous processing based on event triggering frequency. A bifurcated temporal processing module processes the spatiotemporal tensor representation by directly providing temporal features to the cyclic transformer without implementing a multilayer perceptron layer, thereby reducing computational complexity while maintaining feature quality. The cyclic transformer implements three sequential transformer blocks that progressively refine spatiotemporal features through a specialized attention mechanism.

[0056] This method 100 exhibits numerous advantages over traditional object detection methods. Traditional frame-based systems operate at a fixed 30 frames per second, introducing a 33-millisecond processing latency, while this method 100 processes asynchronous event data with microsecond-level temporal resolution. Spatiotemporal tensor representation enables efficient event data processing without the motion blur and dynamic range limitations inherent in frame-based methods. Method 100 overcomes the sparsity limitations of traditional radar and lidar systems through dense event-based sampling, while maintaining computational efficiency through optimized tensor processing and transformer architecture. The coordinate transformation pipeline achieves accurate target localization by integrating temporal variations in object detection, thereby enabling accurate distance estimation and world coordinate determination based on sequential measurement results.

[0057] While the invention has been described in detail with reference to certain embodiments, it should be understood that the invention is not limited to those embodiments. In view of the invention, many modifications and variations can be made by those skilled in the art without departing from the scope of the various embodiments of the invention as described herein. The scope of the invention is therefore indicated by the appended claims rather than by the foregoing description. All changes, modifications, and variations within the meaning and scope of the equivalents of the claims are to be considered within the scope of the claims.

[0058]

Claims

1. A method (100) for determining the coordinates of an object in an environment, characterized in that: Receive asynchronous event data for the environment from the event-based vision sensor (300); A spatiotemporal tensor representation is generated from the received asynchronous event data by sampling a predetermined number of the latest non-zero positions in the time dimension at each spatial location. Temporal features are generated by processing the spatiotemporal tensor representation using a bifurcation time processing module. A cyclic transformer comprising multiple sequential transformer blocks is used to generate spatiotemporal features from the temporal features; The object's position in pixel coordinates is determined based on the generated spatiotemporal features; as well as The coordinate information of the object is determined by converting the determined object position from pixel coordinates to world coordinates.

2. The method (100) according to any one of the preceding claims, wherein generating the spatiotemporal tensor representation comprises: Generate a four-dimensional tensor from the received asynchronous event data, including spatial coordinates, time information, and polarity values; Maintain the temporal order of events within each spatial coordinate location; as well as An adaptive time window is implemented at each spatial coordinate location based on event frequency.

3. The method (100) according to any one of the preceding claims, wherein processing by the bifurcation time processing module includes: The time dimension data of the spatiotemporal tensor representation is processed through a fully connected layer; The processed time-dimensional data is progressively aggregated at adjacent time locations; as well as Weights are assigned to the aggregated data based on their time relevance.

4. The method (100) according to claim 3, wherein the bifurcation time processing module processes the spatiotemporal tensor representation by directly providing the time features to the cyclic transformer without implementing a multilayer perceptron layer.

5. The method (100) according to any one of the preceding claims, wherein generating the spatiotemporal features comprises: The temporal features are processed using a convolutional neural network to arrange pixels in a two-dimensional grid in a computational graph; Pixels are arranged to generate transformed features by transforming the multi-axis self-attention layer; and A self-attention operation is applied to the transformed features to extract global spatial features.

6. The method (100) according to any one of the preceding claims, wherein the cyclic transformer processes the time feature through three sequential transformer blocks, the three sequential transformer blocks comprising: The first transformer block implements spatial self-attention processing for the aforementioned temporal features; A second converter block that performs combined spatial and temporal attention processing on the output from the first converter block; and A third transformer block is used to perform refined temporal attention processing on the output from the second transformer block to generate the spatiotemporal features.

7. The method (100) according to any one of the preceding claims, wherein determining the object location comprises: The generated spatiotemporal features are processed by an event detector configured for high inference speed to generate initial object detection data; as well as The initial object detection data is refined based on the time resolution of the asynchronous event data.

8. The method (100) according to any one of the preceding claims further includes performing event-based data augmentation by: While maintaining the temporal causality, time jitter operation is performed on the asynchronous event data; Performing polarity reversal operations while maintaining spatiotemporal consistency; and Space warping is performed using a perspective transformation matrix while maintaining the temporal relationships of events.

9. The method (100) according to any one of the preceding claims, wherein determining the coordinate information includes: The distance to the detected object is estimated based on the time change of the determined object's location; as well as The world coordinates of the object are determined based on the estimated distance.

10. The method (100) according to any one of the preceding claims, wherein determining the coordinate information further comprises: Perform a perspective transformation to convert the pixel coordinates to initial world coordinates; Linear motion modeling is applied to the initial world coordinates to estimate the trajectory; as well as The world coordinates are refined based on the estimated trajectory.

11. The method (100) according to any one of the preceding claims, wherein the spatiotemporal tensor representation comprises: Event timestamp information; Event pixel coordinates; Event polarity data; as well as Event resolution parameters.

12. The method (100) according to any one of the preceding claims, wherein the spacetime tensor represents: Retain spatial and temporal information from the asynchronous event data; Maintaining polarity; and Asynchronous processing can be implemented based on the frequency of event triggering.

13. A system (200) for determining the coordinates of an object in an environment, comprising: At least one processor; as well as At least one memory, said at least one memory including computer program code for one or more programs, The at least one memory and the computer program code are configured, together with the at least one processor, to cause the system (200) to perform the method (100) according to one or more of claims 1 to 12.

14. A computer program comprising instructions that, when executed by a computer, cause the computer to perform the method (100) according to any one of claims 1 to 12.

15. A computer-readable medium having a computer program thereon as claimed in claim 14.

Images