Asynchronous pulse convolution method and system for integrated bionic visual sensing module

Abstract

This invention provides an asynchronous pulse convolution method and system for an integrated biomimetic visual sensing module, relating to the field of computer vision technology. The method includes acquiring asynchronous event stream data output from an event-driven visual sensor; calculating local activity metrics based on spatiotemporal clustering analysis to divide the data into multiple event regions with different sparsity characteristics; for each event region, adaptively determining the spatial distribution density of sparse convolution kernels and the receptive field shape parameters based on the local activity metrics; extracting spatiotemporal feature representations using an adaptive sparse convolution kernel set, and iteratively optimizing and solving for sparse coding coefficients using a sparse dictionary model; reconstructing visual representation data based on the sparse coding coefficients, and feeding the reconstruction error back to the convolution kernel construction process to form a closed-loop optimization. This invention can adaptively match the sparse distribution characteristics of the event stream, improving feature extraction efficiency and reconstruction accuracy while reducing computational resource consumption.

Description

Technical Field

[0001] This invention relates to computer vision technology, and more particularly to an asynchronous pulse convolution method and system for an integrated bionic vision sensing module that combines sensing, storage, and computing. Background Technology

[0002] With the rapid development of computer vision and artificial intelligence technologies, traditional frame-based vision sensors are gradually revealing their inherent limitations in high-speed motion scenes, low-light environments, and power-sensitive applications. Event-driven vision sensors, as a novel type of biomimetic vision sensor, mimic the perception mechanism of the biological retina, asynchronously capture brightness changes of each pixel in a scene, outputting event stream data with microsecond-level temporal resolution. These sensors offer significant advantages such as high temporal resolution, low latency, high dynamic range, and low power consumption, demonstrating enormous application potential in fields such as autonomous driving, robot vision, and high-speed target tracking.

[0003] The asynchronous event stream data output by event-driven vision sensors differs fundamentally from traditional image data in structure and characteristics, posing new challenges to data processing and feature extraction. Traditional convolutional neural networks, designed for dense, frame-based images, struggle to fully utilize the asynchronous and sparsity characteristics of event data due to their fixed kernel structure and synchronous processing methods. Currently, methods for processing event stream data mainly include event frame transformation methods and methods that directly process the event stream. Event frame transformation methods aggregate asynchronous event streams into synchronous frames for processing, but this approach sacrifices the high temporal resolution advantage of event data. While methods that directly process the event stream preserve the asynchronous nature of the data, improvements are still needed in feature extraction efficiency and accuracy.

[0004] Most existing event stream processing methods employ a uniform convolutional kernel structure to process the entire event stream, failing to adaptively adjust to the varying distribution characteristics of event data across different spatiotemporal regions. Event stream data exhibits significant spatiotemporal sparsity and non-uniform distribution characteristics, with extremely low event density in static background regions and higher event density at the edges of moving targets. Using a fixed convolutional kernel configuration cannot effectively adapt to this dynamically changing data distribution pattern, resulting in a large amount of ineffective computation in sparse regions and the potential loss of crucial information in dense regions due to insufficient receptive fields. This limits both overall computational efficiency and feature extraction quality. Summary of the Invention

[0005] This invention provides an asynchronous pulse convolution method and system for an integrated bionic visual sensing module that combines sensing, storage, and computing, which can solve the problems in the prior art.

[0006] A first aspect of the present invention provides an asynchronous pulse convolution method for an integrated sensing, storage, and computing biomimetic visual sensing module, comprising: Asynchronous event stream data output by the integrated bionic visual sensing module is acquired, spatiotemporal clustering analysis is performed on the asynchronous event stream data, local activity metrics are calculated based on the events in the asynchronous event stream data in the time and space dimensions, spatiotemporal clustering analysis is performed on the asynchronous event stream data according to the local activity metrics, and the asynchronous event stream data is divided into multiple event regions with different sparsity characteristics. For each event region, the spatial distribution density and receptive field shape parameters of the sparse convolution kernels are adaptively determined based on the corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. The asynchronous event stream data is subjected to convolution operation using the adaptive sparse convolution kernel set to extract spatiotemporal feature representations; based on the spatiotemporal feature representations and the preset sparse dictionary model, iterative optimization is performed to obtain sparse coding coefficients by dynamically adjusting the sparse constraint strength and dictionary atom weights according to the reconstruction error in each iteration. The visual representation data of the target scene is reconstructed based on the sparse coding coefficients and the sparse dictionary model; the reconstruction error is fed back to the construction process of the adaptive sparse convolution kernel set to update the receptive field shape parameters and the spatial distribution density, forming a closed-loop optimization.

[0007] Based on the calculation of local activity metrics for events in asynchronous event stream data in both time and spatial dimensions, spatiotemporal clustering analysis is performed on the asynchronous event stream data according to the local activity metrics, dividing the asynchronous event stream data into multiple event regions with different sparsity characteristics, including: For each event in the asynchronous event stream data, extract its spatiotemporal coordinates, count the number of events within the spatial neighborhood within a preset time window, and obtain the time dimension event density and spatial dimension event density corresponding to each event. A spatiotemporal joint density function is constructed, and local activity metrics are calculated by interactively operating the event density in the time dimension and the event density in the spatial dimension based on the spatiotemporal joint density function. The local activity metrics corresponding to each event in the asynchronous event stream data are calculated based on the spatiotemporal joint density function to form a local activity metric distribution map. Using the local activity metric as a clustering feature, a clustering strategy based on density peak detection is adopted to group the asynchronous event stream data. The density peak points in the local activity metric distribution map are identified as cluster centers. Events with similar local activity metrics and continuous spatiotemporal domain are grouped into the same event region, forming multiple event regions with different sparsity characteristics.

[0008] Constructing a spatiotemporal joint density function, and performing interactive calculations on the event density in the time dimension and the event density in the spatial dimension based on the spatiotemporal joint density function to obtain local activity metrics, including: Establish the basic expression form of the spatiotemporal joint density function, which includes a time dimension density term and a space dimension density term. The time dimension density term and the space dimension density term are obtained by substituting the time dimension event density and the space dimension event density into the time variable and the space variable of the spatiotemporal joint density function, respectively. The polarity information of each event in the asynchronous event stream data is extracted, and the polarity gradient feature is calculated based on the change pattern of the polarity information in the spatiotemporal domain. The polarity gradient feature is then used to modulate the weight coefficients of the time variables and spatial variables in the spatiotemporal joint density function. The modulated time-dimensional density term and the spatial-dimensional density term are used as inputs, and the time-dimensional density term and the spatial-dimensional density term are interactively operated on by a nonlinear coupling operator. The nonlinear coupling operator determines the coupling strength based on the mutual information between the time-dimensional density term and the spatial-dimensional density term. The local activity metric value corresponding to each event is calculated based on the operation result of the nonlinear coupling operator.

[0009] For each event region, the spatial distribution density and receptive field shape parameters of the sparse convolution kernels are adaptively determined based on their corresponding local activity metrics, thereby constructing an adaptive set of sparse convolution kernels that matches the event distribution characteristics, including: Calculate the statistical distribution characteristics of the local activity metrics of all events within each event region, and substitute the statistical distribution characteristics into a preset density mapping function to establish a mapping relationship between the local activity metrics and the spatial distribution density of sparse convolution kernels; Based on the mapping relationship, the local activity metric is converted into a sparse convolution kernel spatial distribution density; The spatial diffusion degree of the event distribution is calculated based on the variance characteristics of the local activity metric, and principal component analysis is performed on the spatial distribution of the event using the spatial diffusion degree. The principal direction vector is extracted as the major axis direction in the receptive field shape parameter based on the results of the principal component analysis, and the major axis dimension in the receptive field shape parameter is determined based on the spatial diffusion degree. Based on the spatial distribution density of the sparse convolution kernels and the receptive field shape parameters, a corresponding number of sparse convolution kernels are generated within the event region; the sparse convolution kernels generated in all event regions are summarized to construct an adaptive sparse convolution kernel set that matches the event distribution characteristics.

[0010] Based on the spatiotemporal feature representation and the preset sparse dictionary model, iterative optimization is performed. In each iteration, the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error to obtain sparse coding coefficients, including: Initialize the sparse dictionary model, take the spatiotemporal feature representation as input data, use the sparse dictionary model to perform sparse decomposition on the spatiotemporal feature representation, and solve the initial sparse coding coefficients under sparse constraints. Based on the initial sparse coding coefficients and the sparse dictionary model, reconstruct the spatiotemporal domain features, and calculate the reconstruction error between the reconstructed spatiotemporal domain features and the spatiotemporal domain feature representation; Based on the reconstruction error, calculate the sparse constraint strength adjustment and dictionary atom weight adjustment. Update the sparse constraint strength using the sparse constraint strength adjustment and update the weight value of each dictionary atom in the sparse dictionary model using the dictionary atom weight adjustment. Based on the updated sparse constraint strength and the updated sparse dictionary model, a new round of sparse decomposition is performed on the spatiotemporal feature representation to obtain updated sparse coding coefficients; a new reconstruction error is calculated based on the updated sparse coding coefficients, and it is determined whether the new reconstruction error meets the convergence condition. If the convergence condition is met, the current sparse coding coefficients are output.

[0011] Initialize the sparse dictionary model, taking the spatiotemporal feature representation as input data, and perform sparse decomposition on the spatiotemporal feature representation using the sparse dictionary model. Solving for the initial sparse coding coefficients under sparse constraints includes: Feature samples are extracted from the spatiotemporal domain feature representation, and clustering is performed on the feature samples. An initial set of dictionary atoms for the sparse dictionary model is constructed based on the cluster centers. An initial weight value is assigned to each dictionary atom in the initial set of dictionary atoms to complete the initialization of the sparse dictionary model. Using the spatiotemporal feature representation as input data, the spatiotemporal feature representation is linearly represented by the dictionary atom set in the sparse dictionary model, and the correlation between the spatiotemporal feature representation and the sparse coding coefficients is established. A sparse constraint is set, which limits the upper limit of the number of non-zero elements in the sparse coding coefficients. The sparse constraint is introduced into the sparse decomposition process as a solution constraint. Under the constraint of the sparse constraint, the initial sparse coding coefficients that satisfy the sparse constraint are solved by optimizing the error between the spatiotemporal domain feature representation and the reconstruction result of the sparse dictionary model.

[0012] Feeding the reconstruction error back to the construction process of the adaptive sparse convolution kernel set, and updating the receptive field shape parameters and the spatial distribution density, includes: The density update and shape update are calculated based on the reconstruction error of the current iteration; when the reconstruction error is greater than a preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is increased and / or the receptive field major axis size is increased; when the reconstruction error is less than the preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is decreased and / or the receptive field major axis size is reduced. The major axis direction and / or major axis dimension in the receptive field shape parameters are updated based on the shape update amount, and boundary constraints are applied to the updated receptive field shape parameters. The adaptive sparse convolution kernel set is reconstructed using the updated spatial density and updated receptive field shape parameters, and then proceeds to the next iteration.

[0013] Feeding the reconstruction error back to the construction process of the adaptive sparse convolution kernel set, and updating the receptive field shape parameters and the spatial distribution density, includes: The density update and shape update are calculated based on the reconstruction error of the current iteration; when the reconstruction error is greater than a preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is increased and / or the receptive field major axis size is increased; when the reconstruction error is less than the preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is decreased and / or the receptive field major axis size is reduced. The major axis direction and / or major axis dimension in the receptive field shape parameters are updated based on the shape update amount, and boundary constraints are applied to the updated receptive field shape parameters. The adaptive sparse convolution kernel set is reconstructed using the updated spatial density and updated receptive field shape parameters, and then proceeds to the next iteration.

[0014] A second aspect of the present invention provides an asynchronous pulse convolution system for an integrated biomimetic visual sensing module, comprising: The acquisition module is used to acquire asynchronous event stream data output by the integrated bionic visual sensing module, perform spatiotemporal clustering analysis on the asynchronous event stream data, calculate local activity metrics based on the events in the asynchronous event stream data in the time and space dimensions, perform spatiotemporal clustering analysis on the asynchronous event stream data according to the local activity metrics, and divide the asynchronous event stream data into multiple event regions with different sparsity characteristics. The module is used to adaptively determine the spatial distribution density and receptive field shape parameters of sparse convolution kernels for each event region based on the corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. The convolution module is used to perform convolution operations on the asynchronous event stream data using the adaptive sparse convolution kernel set to extract spatiotemporal feature representations; iterative optimization is performed based on the spatiotemporal feature representations and the preset sparse dictionary model, and the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error in each iteration to obtain sparse coding coefficients; The closed-loop module is used to reconstruct the visual representation data of the target scene based on the sparse coding coefficients and the sparse dictionary model; feed the reconstruction error back to the construction process of the adaptive sparse convolution kernel set, update the receptive field shape parameters and the spatial distribution density, and form a closed-loop optimization.

[0015] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0016] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0017] The beneficial effects of this application are as follows: This invention effectively identifies event regions with different sparsity characteristics by performing spatiotemporal clustering analysis on asynchronous event stream data and calculating local activity metrics, thereby achieving refined partitioning of asynchronous event stream data. This adaptive partitioning strategy based on event distribution density fully utilizes the sparsity characteristics of event-driven visual sensor output data, avoids unnecessary calculations on redundant regions, significantly reduces computational complexity and storage overhead in data processing, and improves the overall processing efficiency of the system.

[0018] This invention adaptively determines the spatial distribution density and receptive field shape parameters of sparse convolution kernels based on local activity metrics, constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. This achieves dynamic adaptation between the convolution kernel structure and the input data characteristics. Compared to traditional fixed convolution kernel methods, this adaptive mechanism can employ optimal feature extraction strategies for event regions with different sparsity characteristics, significantly reducing computational load while ensuring feature extraction quality, and improving the accuracy and robustness of spatiotemporal feature representation. Attached Figure Description

[0019] Figure 1 This is a flowchart illustrating the asynchronous pulse convolution method of the integrated sensing, storage, and computing biomimetic visual sensing module according to an embodiment of the present invention. Detailed Implementation

[0020] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0021] The technical solution of the present invention will be described in detail below with reference to specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments.

[0022] Figure 1 This is a flowchart illustrating the asynchronous pulse convolution method of the integrated sensing, storage, and computing biomimetic vision sensing module according to an embodiment of the present invention. Figure 1 As shown, the method includes: Asynchronous event stream data output by the integrated bionic visual sensing module is acquired, spatiotemporal clustering analysis is performed on the asynchronous event stream data, local activity metrics are calculated based on the events in the asynchronous event stream data in the time and space dimensions, spatiotemporal clustering analysis is performed on the asynchronous event stream data according to the local activity metrics, and the asynchronous event stream data is divided into multiple event regions with different sparsity characteristics. For each event region, the spatial distribution density and receptive field shape parameters of the sparse convolution kernels are adaptively determined based on the corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. The asynchronous event stream data is subjected to convolution operation using the adaptive sparse convolution kernel set to extract spatiotemporal feature representations; based on the spatiotemporal feature representations and the preset sparse dictionary model, iterative optimization is performed to obtain sparse coding coefficients by dynamically adjusting the sparse constraint strength and dictionary atom weights according to the reconstruction error in each iteration. The visual representation data of the target scene is reconstructed based on the sparse coding coefficients and the sparse dictionary model; the reconstruction error is fed back to the construction process of the adaptive sparse convolution kernel set to update the receptive field shape parameters and the spatial distribution density, forming a closed-loop optimization.

[0023] For example, the integrated sensing, storage, and computing bionic vision sensing module employs a 128×128 pixel dynamic vision sensor array, with each pixel unit integrating a photoelectric conversion layer and an analog computing layer. The photoelectric conversion layer uses InGaAs material and has a response time of 1 microsecond, triggering an asynchronous event when the incident light intensity changes by more than a 15% threshold. The analog computing layer performs convolution operations through current-mode multiply-accumulate operations, consuming only 0.3mW.

[0024] In one optional implementation, local activity metrics are calculated based on events in the asynchronous event stream data in both time and spatial dimensions. Spatiotemporal clustering analysis is then performed on the asynchronous event stream data based on these local activity metrics, dividing the asynchronous event stream data into multiple event regions with different sparsity characteristics, including: For each event in the asynchronous event stream data, extract its spatiotemporal coordinates, count the number of events within the spatial neighborhood within a preset time window, and obtain the time dimension event density and spatial dimension event density corresponding to each event. A spatiotemporal joint density function is constructed, and local activity metrics are calculated by interactively operating the event density in the time dimension and the event density in the spatial dimension based on the spatiotemporal joint density function. The local activity metrics corresponding to each event in the asynchronous event stream data are calculated based on the spatiotemporal joint density function to form a local activity metric distribution map. Using the local activity metric as a clustering feature, a clustering strategy based on density peak detection is adopted to group the asynchronous event stream data. The density peak points in the local activity metric distribution map are identified as cluster centers. Events with similar local activity metrics and continuous spatiotemporal domain are grouped into the same event region, forming multiple event regions with different sparsity characteristics.

[0025] When performing spatiotemporal clustering analysis on asynchronous event stream data, an event data preprocessing module is established. This module receives the raw asynchronous event stream from an event-driven vision sensor. Each event contains four-dimensional information: horizontal coordinate x, vertical coordinate y, timestamp t, and polarity p. The preprocessing module standardizes the format of the input event stream, ensuring that the coordinate accuracy reaches the pixel level, the timestamp accuracy is at the microsecond level, and the polarity value is limited to +1 or -1. The data cache adopts a circular queue structure with a default capacity of 100,000 events, supporting real-time overwriting to avoid memory overflow.

[0026] During the event spatiotemporal coordinate extraction process, the preprocessed event stream data is traversed, and the spatial coordinates (xi, yi) and temporal coordinates (ti) are extracted for each event ei. The spatial coordinates are read directly from the event data, and their value range is determined based on the sensor resolution, commonly 640×480 pixels or 1280×720 pixels. The temporal coordinates are represented using relative time, with the start time of the data stream as zero, and the unit is microseconds. The coordinate extraction module maintains an event index table, establishing a mapping relationship between event sequence numbers and their spatiotemporal coordinates. The index table uses a hash structure to support fast lookups.

[0027] The time-dimensional event density calculation employs a sliding window strategy. The default time window length is 10 milliseconds, which can be dynamically adjusted between 1 and 100 milliseconds depending on the scenario. For the currently processed event *ei*, the system counts the total number of events within the time window [*ti-Δt / 2*, *ti+Δt / 2*] to obtain the time-dimensional event density ρt(*ei*). The sliding window uses an overlapping strategy, with an overlap ratio set to 50% to ensure the continuity of density calculation. The time density calculation module maintains a time-series event queue, sorted in ascending order by timestamp, and supports binary search to improve statistical efficiency.

[0028] Spatial event density calculation is based on spatial neighborhood statistics. The neighborhood radius is set to 5 pixels by default and can be adjusted within the range of 2 to 20 pixels. For the spatial coordinates (xi, yi) of event ei, the system counts the number of events within the same time window within a circular region centered at these coordinates and with a radius of r, obtaining the spatial event density ρs(ei). The spatial density calculation uses a grid index structure, dividing the sensor plane into several grid cells. The side length of each cell is set to half the neighborhood radius. Events are assigned to the corresponding grid cells according to their spatial coordinates, and only the 9 adjacent grid cells need to be searched during neighborhood search.

[0029] The spatiotemporal joint density function is constructed using a nonlinear coupling approach, fusing the event densities of the temporal and spatial dimensions. The joint density function employs a weighted geometric mean, with weighting coefficients αt and αs corresponding to the importance of the temporal and spatial densities, respectively. Both are set to a default value of 0.5, satisfying the normalization constraint αt + αs = 1. A logarithmic transformation is used in the nonlinear mapping to compress the dynamic range and avoid excessive influence of extreme values ​​on the results. The joint density calculation module receives the temporal and spatial densities as input and outputs initial values ​​for the local activity metric, with the numerical range limited to 0 to 1 through min-max normalization.

[0030] During the calculation of local activity metrics, a spatiotemporal joint density function is applied to each event to obtain the corresponding local activity metric. The calculation employs an event-by-event traversal approach, maintaining an array of metrics where the array index corresponds to the event number. The metric calculation considers a time decay factor; events further removed from the current time are affected by exponential decay in their metrics. The decay factor is set to 0.95 by default and can be adjusted within the range of 0.8 to 0.99. The metric array supports parallel computation, employing multi-threaded processing to improve computational efficiency. The number of threads is automatically configured based on the number of processor cores.

[0031] The local activity metric distribution map generation process maps the calculated metric values ​​to the sensor spatial coordinate system, forming a two-dimensional distribution image. The distribution map is stored using a floating-point matrix with a size consistent with the sensor resolution. For pixel locations with multiple events, overlapping events are handled by metric accumulation. The distribution map supports multi-scale representation and is smoothed using a Gaussian filter with a kernel standard deviation set to 2 pixels and a filter radius three times the kernel standard deviation. The distribution map data structure supports sparse storage, only storing the coordinates and metric values ​​of non-zero pixels to save memory space.

[0032] The density peak detection clustering strategy identifies cluster centers based on the local activity metric distribution map. Peak detection employs a local maximum search algorithm, comparing metric values ​​within the 8-neighborhood of each pixel. Pixels satisfying the local maximum condition and whose metric values ​​exceed a preset threshold are identified as candidate cluster centers. The threshold setting uses an adaptive strategy, dynamically adjusting based on the statistical characteristics of the distribution map, with a default value of the distribution map mean plus one standard deviation. Candidate cluster centers are further filtered through a significance test, requiring that the gradient of the metric values ​​within a certain radius around the peak point satisfy a monotonically decreasing characteristic; the radius parameter is set to 3 pixels by default.

[0033] The event grouping process employs a region-growing strategy. Starting from each confirmed cluster center, events with similar local activity metrics and spatiotemporal continuity are grouped into the same event region. Metric similarity is judged using a relative error standard; two events are considered similar if the relative difference in their metrics is less than 10%. Spatiotemporal continuity is determined by requiring events to be spatially adjacent pixels with a time interval less than a preset threshold, which is set to half the length of the time window by default. Region growing uses a breadth-first search strategy, maintaining a queue of candidate events and expanding layer by layer from the cluster center until no adjacent events meet the criteria.

[0034] Event region labeling uses integer numbering, with each region assigned a unique identifier, and events within the same region sharing the same identifier. Region characteristic statistics include parameters such as the number of events, average metric, time span, and spatial range. Sparsity quantification is based on statistical indicators of event density distribution within the region, including higher-order moment features such as variance, skewness, and kurtosis. The region management module maintains a list of regions and supports region merging and splitting operations. Merging is performed when the characteristic differences between adjacent regions are less than a preset threshold, and splitting is performed when a region exhibits a clear bimodal distribution.

[0035] In one optional implementation, a spatiotemporal joint density function is constructed, and a local activity metric is calculated by interactively operating the event density in the time dimension and the event density in the spatial dimension based on the spatiotemporal joint density function, including: Establish the basic expression form of the spatiotemporal joint density function, which includes a time dimension density term and a space dimension density term. The time dimension density term and the space dimension density term are obtained by substituting the time dimension event density and the space dimension event density into the time variable and the space variable of the spatiotemporal joint density function, respectively. The polarity information of each event in the asynchronous event stream data is extracted, and the polarity gradient feature is calculated based on the change pattern of the polarity information in the spatiotemporal domain. The polarity gradient feature is then used to modulate the weight coefficients of the time variables and spatial variables in the spatiotemporal joint density function. The modulated time-dimensional density term and the spatial-dimensional density term are used as inputs, and the time-dimensional density term and the spatial-dimensional density term are interactively operated on by a nonlinear coupling operator. The nonlinear coupling operator determines the coupling strength based on the mutual information between the time-dimensional density term and the spatial-dimensional density term. The local activity metric value corresponding to each event is calculated based on the operation result of the nonlinear coupling operator.

[0036] In establishing the basic expression of the spatiotemporal joint density function, the density function module adopts a bivariate function structure, including time and spatial variables as independent input parameters. The time dimension density term is obtained by substituting the time dimension event density into the position of the time variable, and the spatial dimension density term is obtained by substituting the spatial dimension event density into the position of the spatial variable. The basic expression adopts a weighted combination structure: the time dimension density term is multiplied by a time weight coefficient, and the spatial dimension density term is multiplied by a spatial weight coefficient; the sum of the two constitutes the basic form of the joint density function. The initial values ​​of both the time and spatial weight coefficients are set to 0.5, satisfying the normalization constraint that the sum of the time and spatial weight coefficients equals 1. The density function module maintains a parameter configuration table, supporting dynamic adjustment of the weight coefficients, with the adjustment range limited to 0.1 to 0.9 and an adjustment precision of 0.01.

[0037] The polarity information extraction process reads the polarity identifier of each event from the asynchronous event stream data. Polarity values ​​are limited to positive 1 for an increase in brightness or negative 1 for a decrease in brightness. Polarity information is stored in a compressed format, using a single bit to represent the polarity state, reducing storage overhead. The polarity extraction module traverses the event stream sequence, establishing a mapping relationship between event indices and polarity values. The mapping table uses a bit vector structure to support fast lookups and batch operations. The validity of polarity information is verified through statistical checks, requiring the ratio of positive to negative polarity events to be within the range of 0.2 to 0.8. Data exceeding this range is marked as abnormal and triggers an early warning mechanism.

[0038] Polar gradient feature calculation is based on the analysis of polarity information change patterns in the spatiotemporal domain. The time-domain polar gradient is obtained by calculating the rate of polarity change within adjacent time windows, with a time window length of 5 milliseconds and a window sliding step of 1 millisecond. The spatial-domain polar gradient is obtained by calculating the difference in polarity distribution within an 8-neighborhood of the event location, using a gradient operator template for gradient calculation. The polar gradient feature includes two components: magnitude and direction. The magnitude represents the intensity of the polarity change, and the direction represents the dominant direction of the polarity change. The gradient feature calculation module adopts a parallel processing architecture, supporting multi-threaded computation to improve processing efficiency. The thread pool size is configured according to the number of processor cores, with a default of 4 worker threads.

[0039] The weighting coefficient modulation process dynamically adjusts the weights of time and spatial variables in the spatiotemporal joint density function using polar gradient characteristics. The modulation strategy determines the modulation intensity based on the magnitude of the polar gradient; the larger the gradient magnitude, the greater the increase in the weight coefficient of the corresponding dimension. The time-dimensional weight modulation is obtained by multiplying the time-domain polar gradient magnitude by a time modulation factor, and the spatial-dimensional weight modulation is obtained by multiplying the spatial-domain polar gradient magnitude by a spatial modulation factor. The default value for both the time and spatial modulation factors is 0.1, and they can be adjusted within the range of 0.05 to 0.3. The modulated weight coefficients need to be renormalized to maintain the constraint that the weight sum is 1; normalization is achieved using linear scaling.

[0040] The calculation of the modulated time and spatial density terms applies the modulated weighting coefficients to the basic density terms. The modulated time density term equals the time-dimensional event density multiplied by the modulated time weighting coefficient, and the modulated spatial density term equals the spatial event density multiplied by the modulated spatial weighting coefficient. The density term calculation module maintains an intermediate result cache, employing a double-buffering mechanism to avoid data contention during calculation. The cache capacity is set to twice the number of currently processed events, supporting fast access to historical data. The density term values ​​are dynamically normalized to a range of 0 to 1, with the normalization parameter updated based on statistical characteristics within a sliding window of 1000 events.

[0041] The design of the nonlinear coupling operator is based on the calculation of mutual information between the time-dimensional and spatial-dimensional density terms. The mutual information is estimated using a discretization method, dividing the numerical space of the density terms into 256 equally spaced intervals and calculating the joint and marginal probability distributions. The mutual information is calculated using the log-likelihood ratio method, quantifying the correlation between the two density terms by statistically analyzing the deviation between the joint and independent distributions. The coupling strength parameter is determined based on the magnitude of the mutual information; the larger the mutual information, the stronger the coupling. The value of the coupling strength parameter ranges from 0.1 to 2.0, with a default value of 1.0.

[0042] The interactive computation process fuses the modulated time and spatial density terms using a nonlinear coupling operator. The coupling operator employs a power function, raising both the time and spatial density terms to powers of the coupling strength parameter, followed by a geometric mean calculation. The interactive computation module uses a lookup table to calculate the power function, pre-calculating the results for commonly used exponent values ​​to reduce real-time computation overhead. The lookup table has a precision of 0.001, covering exponent values ​​from 0 to 2, and a size of 2000 entries. The computation results are smoothed using a 5-point moving average window to reduce noise.

[0043] The calculation of local activity metrics uses the results of nonlinear coupling operators as the raw metrics, which are then post-processed to obtain the final local activity metrics. Post-processing includes two steps: outlier detection and numerical stabilization. Outlier detection uses a three-standard-deviation criterion; values ​​exceeding the mean plus or minus three standard deviations are identified as outliers and corrected using a median replacement strategy. Numerical stabilization is achieved through logarithmic transformation, mapping the metrics to logarithmic space to compress the dynamic range and avoid the influence of extreme values ​​on subsequent processing. The metric calculation module supports batch processing, with the number of events processed per batch configurable between 100 and 10000, with a default setting of 1000 events.

[0044] The output format of the metrics uses a floating-point array structure, with the array length equal to the number of input events, and the array index corresponding to the event sequence number. The output values ​​maintain single-precision floating-point precision, with the value range normalized to between 0 and 1. Metric data supports multiple storage formats, including binary format for efficient storage and text format for debugging and analysis. Data integrity is ensured through a checksum and verification mechanism, employing a cyclic redundancy check algorithm to calculate data checksums and detect data errors during transmission and storage.

[0045] In one optional implementation, for each event region, the spatial distribution density and receptive field shape parameters of the sparse convolution kernels are adaptively determined based on their corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics, including: Calculate the statistical distribution characteristics of the local activity metrics of all events within each event region, and substitute the statistical distribution characteristics into a preset density mapping function to establish a mapping relationship between the local activity metrics and the spatial distribution density of sparse convolution kernels; Based on the mapping relationship, the local activity metric is converted into a sparse convolution kernel spatial distribution density; The spatial diffusion degree of the event distribution is calculated based on the variance characteristics of the local activity metric, and principal component analysis is performed on the spatial distribution of the event using the spatial diffusion degree. The principal direction vector is extracted as the major axis direction in the receptive field shape parameter based on the results of the principal component analysis, and the major axis dimension in the receptive field shape parameter is determined based on the spatial diffusion degree. Based on the spatial distribution density of the sparse convolution kernels and the receptive field shape parameters, a corresponding number of sparse convolution kernels are generated within the event region; the sparse convolution kernels generated in all event regions are summarized to construct an adaptive sparse convolution kernel set that matches the event distribution characteristics.

[0046] The event region statistical distribution feature calculation module iterates through all events within each event region, extracts the corresponding local activity metrics for each event, and constructs a metric array. The statistical distribution feature calculation includes parameters such as mean, variance, skewness, and kurtosis. The mean reflects the average level of event activity within the region, the variance reflects the dispersion of activity metrics, the skewness reflects the asymmetry of the distribution, and the kurtosis reflects the sharpness of the distribution. The statistical calculation employs an incremental update strategy; when a new event is added to the region, the statistical parameters are updated using the Welford algorithm to avoid repeatedly calculating the entire array. The statistical precision is set to double-precision floating-point numbers to ensure the numerical stability of the calculation results, and the statistical results retain 6 significant digits.

[0047] During the establishment of the density mapping function, the preset mapping function adopts a piecewise linear interpolation form, dividing the local activity metric domain into 10 equally spaced intervals, each interval corresponding to a different spatial distribution density range. The input of the mapping function is the statistical distribution characteristics of the event region, and the output is the spatial distribution density parameters of the sparse convolution kernel. The mapping relationship is based on the mean of the metric as the main mapping variable and the variance as the modulation factor; the larger the mean of the metric, the higher the corresponding spatial distribution density. The parameter configuration of the mapping function adopts a lookup table structure, with each entry including the metric threshold, the basic density value, and the modulation coefficient. The table size is 100 entries, supporting binary search to improve retrieval efficiency.

[0048] The sparse convolution kernel spatial distribution density transformation process substitutes statistical distribution characteristics into the density mapping function to calculate the spatial distribution density value corresponding to each event region. The spatial distribution density value is limited to the range of 0.1 to 10.0, with the unit being the convolution kernel per square pixel, and the numerical precision is maintained at 0.01. The density transformation module maintains a region density mapping table, recording the relationship between each event region identifier and its corresponding density value. The mapping table adopts a hash table structure to support fast lookup. Boundary checks of density values ​​ensure that the transformation results are within the valid range; values ​​outside the range are truncated, and anomaly logs are recorded for subsequent analysis.

[0049] The total number of convolutional kernels is calculated based on the spatial distribution density of sparse convolutional kernels and the spatial area of ​​the event region. The event region area is obtained by statistically analyzing the spatial coordinate range of events within the region. The width and height of the bounding box are calculated, and the area equals the width multiplied by the height. The total number of convolutional kernels equals the spatial distribution density multiplied by the region area. The calculation result is rounded up to ensure complete coverage of the region. The kernel count calculation module sets minimum and maximum limits for the number of convolutional kernels: a minimum of 5 and a maximum of 1000. If the limits are exceeded, the spatial distribution density is adjusted to meet the constraints.

[0050] Spatial diffusion degree calculation quantifies the spatial dispersion of event distribution by utilizing the variance characteristics of local activity metrics. Variance calculation employs the sample variance formula, with the denominator being the number of events minus 1, ensuring unbiased estimation. Spatial diffusion degree is defined as the square root of the variance, reflecting the spatial distribution range of events. The numerical range of diffusion degree is limited to 0 to 1 through normalization, with the normalization parameter determined based on the statistical characteristics of the global event distribution. The diffusion degree calculation module supports real-time updates; when events within the event region change, the diffusion degree value is updated incrementally.

[0051] Principal component analysis (PCA) performs dimensionality reduction on the spatial coordinates of events within the event region, extracting the main directions of change. The spatial coordinate data is first centered, subtracting the mean to obtain zero-mean data. The covariance matrix is ​​calculated using a 2×2 matrix, containing the variances of the x-coordinate, y-coordinate, and coordinate covariance. Eigenvalue decomposition employs the Jacobi iterative algorithm to calculate the eigenvalues ​​and eigenvectors of the covariance matrix, with an iteration precision of 0.0001 and a maximum of 100 iterations. The PCA module returns the eigenvector corresponding to the first principal component as the principal direction vector, with the magnitude of the eigenvector normalized to 1.

[0052] The principal direction vector is extracted as the major axis direction in the receptive field shape parameters. This vector contains both horizontal and vertical coordinate components, with values ​​ranging from -1 to 1. The angle of the principal direction vector is represented in radians, calculated using the arctangent function to determine the angle between the vector and the horizontal axis. The vector extraction module performs a validity check, ensuring the vector magnitude is close to 1; if the deviation exceeds 0.01, it is recalculated. The major axis direction is stored as a unit vector to reduce storage space and simplify subsequent calculations.

[0053] The major axis dimension is determined based on the magnitude of spatial diffusion; the greater the diffusion, the longer the major axis dimension. The major axis dimension is calculated using a linear mapping relationship, multiplying the diffusion level by a size magnification factor to obtain the major axis dimension per pixel. The default value for the size magnification factor is 20, which can be adjusted between 10 and 50, based on the average size of the event region. The value range of the major axis dimension is limited to 3 to 100 pixels to ensure that the receptive field captures local features without overexpansion. The minor axis dimension is obtained by multiplying the major axis dimension by the aspect ratio, which is determined based on the ratio of the second eigenvalue to the first eigenvalue in principal component analysis.

[0054] The sparse convolution kernel generation process configures a corresponding number of convolution kernels within each event region according to the spatial distribution density and receptive field shape parameters. The convolution kernel positions employ a uniform distribution strategy, dividing the region space into equal grids and placing the kernel at the center of each grid. The receptive field shape of the convolution kernel is elliptical, with the major axis aligned with the principal direction vector; the major and minor axis dimensions are determined based on the shape parameters. The convolution kernel weights are initialized using a Gaussian distribution, with the distribution center located at the center of the receptive field and a standard deviation of 1 / 3 of the major axis dimension. The generation module maintains a list of convolution kernel parameters, including position coordinates, receptive field parameters, and weight arrays.

[0055] The kernel aggregation process collects sparse convolutional kernels generated from all event regions, constructing a global adaptive sparse convolutional kernel set. Aggregation uses an append-only approach, adding kernels to the set sequentially according to the region processing order. The kernel set employs a dynamic array structure, supporting automatic expansion to adapt to different data sizes. Each kernel in the set is assigned a unique identifier containing both the region number and its intra-kernel sequence number, facilitating subsequent indexing and management. The aggregation module performs duplicate checks to prevent the generation of multiple similar kernels at the same location.

[0056] In one optional implementation, iterative optimization is performed based on the spatiotemporal feature representation and a preset sparse dictionary model. In each iteration, the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error to obtain sparse coding coefficients including: Initialize the sparse dictionary model, take the spatiotemporal feature representation as input data, use the sparse dictionary model to perform sparse decomposition on the spatiotemporal feature representation, and solve the initial sparse coding coefficients under sparse constraints. Based on the initial sparse coding coefficients and the sparse dictionary model, reconstruct the spatiotemporal domain features, and calculate the reconstruction error between the reconstructed spatiotemporal domain features and the spatiotemporal domain feature representation; Based on the reconstruction error, calculate the sparse constraint strength adjustment and dictionary atom weight adjustment. Update the sparse constraint strength using the sparse constraint strength adjustment and update the weight value of each dictionary atom in the sparse dictionary model using the dictionary atom weight adjustment. Based on the updated sparse constraint strength and the updated sparse dictionary model, a new round of sparse decomposition is performed on the spatiotemporal feature representation to obtain updated sparse coding coefficients; a new reconstruction error is calculated based on the updated sparse coding coefficients, and it is determined whether the new reconstruction error meets the convergence condition. If the convergence condition is met, the current sparse coding coefficients are output.

[0057] In implementing this invention, the initialization process of the sparse dictionary model is constructed using a dictionary matrix of a preset dimension. The number of rows in this dictionary matrix matches the feature dimension of the spatiotemporal domain feature representation, and the number of columns corresponds to the number of dictionary atoms. Specifically, when the feature dimension of the spatiotemporal domain feature representation is 512-dimensional, the number of rows in the dictionary matrix is ​​set to 512, and the number of dictionary atoms is set to 1024, thus forming a 512 x 1024 dictionary matrix. The values ​​of each element in the dictionary matrix are filled with random values ​​following a standard normal distribution generated by a random number generator. Subsequently, each dictionary atom is normalized so that the modulus of each atom is equal to 1. The spatiotemporal domain feature representation is used as the input data to be decomposed. This input data is organized into a matrix according to the number of samples and the feature dimension. For example, the input data matrix containing 100 samples has a dimension of 512 x 100.

[0058] The sparse decomposition process obtains the initial sparse coding coefficients by solving an optimization problem that requires minimizing the reconstruction error while satisfying sparsity constraints. Specifically, the reconstruction error is obtained by calculating the difference between the product of the dictionary matrix and the sparse coding coefficient matrix and the input data matrix, using the sum of squared element-wise differences. The sparsity constraint is achieved by limiting the number of non-zero elements in the sparse coding coefficient matrix, with the initial sparsity constraint strength parameter set to 0.1. An iterative threshold shrinkage algorithm is used during the solution process. This algorithm iteratively updates the values ​​of each element in the sparse coding coefficient matrix, setting elements with coefficient values ​​less than a threshold to zero in each iteration. The threshold value is determined by the sparsity constraint strength parameter. After multiple iterations, iteration stops when the coefficient change between two consecutive iterations is less than 0.001, resulting in the initial sparse coding coefficient matrix, which has dimensions of 1024 x 100.

[0059] The calculation of reconstructed spatiotemporal features involves matrix multiplication of the dictionary matrix with the initial sparse coding coefficient matrix. The result of this multiplication generates the reconstructed spatiotemporal feature matrix, which has the same dimensions as the original input data matrix (512 x 100). The reconstruction error is calculated element-wise by subtracting the corresponding elements of the reconstructed spatiotemporal feature matrix from those of the original spatiotemporal feature representation matrix to obtain the error matrix. The average reconstruction error is obtained by summing the squares of all elements in the error matrix and then dividing by the total number of elements. In a specific case, the average reconstruction error obtained in a certain iteration was 2.35.

[0060] The calculation of the sparse constraint strength adjustment is based on the changing trend of the reconstruction error value. When the reconstruction error value exceeds the preset target error threshold, the sparse constraint strength needs to be reduced to lower the reconstruction error. The adjustment amount is calculated by multiplying the difference between the reconstruction error value and the target error threshold by an adjustment coefficient. The target error threshold is set to 1.5, and the adjustment coefficient is set to 0.05. When the reconstruction error value is 2.35, the difference is calculated to be 0.85. Multiplying this by the adjustment coefficient yields a sparse constraint strength adjustment of 0.0425. This adjustment amount is subtracted from the current sparse constraint strength value, which is currently 0.1. The updated sparse constraint strength value is 0.0575.

[0061] The calculation of dictionary atom weight adjustment is performed individually for each atom in the dictionary matrix. For a specific dictionary atom, the number of times that atom is activated in the sparse coding of all samples is counted, i.e., the number of non-zero elements in the sparse coding coefficients of the corresponding row. Simultaneously, the local reconstruction error generated when this atom participates in reconstruction is calculated. This error is obtained by comparing the product of the atom and its corresponding sparse coding coefficient with the corresponding part of the original feature. When an atom has a low activation count and a large local reconstruction error, its weight value needs to be reduced. The weight adjustment is calculated by multiplying the inverse of the activation count by the local reconstruction error and dividing by a normalization factor. In a specific case, dictionary atom number 58 is activated 15 times in 100 samples, with a local reconstruction error of 0.8. With a normalization factor set to 50, the weight adjustment for this atom is calculated as the inverse of the activation count (15), 0.0667, multiplied by the error (0.8), and then divided by the normalization factor 50, resulting in 0.001067. The original weight value was 1.0, and the updated weight value was 0.998933.

[0062] The dictionary matrix is ​​updated by applying the corresponding weight value to each atom. Each column element of the dictionary matrix is ​​multiplied by the updated weight value of the atom in that column. After updating the weights of all atoms, each atom is normalized again to ensure that the modulus of each atom remains 1. The updated dictionary matrix retains its original dimension of 512 x 1024.

[0063] The new round of sparse decomposition uses updated sparse constraint strength values ​​and an updated dictionary matrix. The calculation method is the same as the initial sparse decomposition process, but the parameters have been adjusted. Since the sparse constraint strength is reduced to 0.0575, the threshold in the iterative shrinkage threshold algorithm is correspondingly reduced, allowing more coding coefficients to remain non-zero. After iterative calculation, an updated sparse coding coefficient matrix is ​​obtained, in which the number of non-zero elements increases compared to the initial sparse coding coefficients.

[0064] The new reconstruction error is calculated using the same method as the initial reconstruction error. The updated dictionary matrix is ​​multiplied by the updated sparse coding coefficient matrix to obtain the new reconstructed spatiotemporal feature matrix, which is then compared with the original spatiotemporal feature representation to calculate the average reconstruction error. In this iteration, the new average reconstruction error value is reduced to 1.68.

[0065] The convergence condition is determined by comparing the new reconstruction error with a preset convergence threshold of 1.5. If the new reconstruction error value of 1.68 is still greater than this threshold, the convergence condition is not met, and the next round of iteration optimization is required. In the next iteration, the steps of reconstruction error calculation, parameter adjustment, and sparse decomposition are repeated. After multiple iterations, when the reconstruction error value calculated in a certain iteration decreases to 1.42, which is less than the convergence threshold of 1.5, the convergence condition is met, and the current sparse coding coefficient matrix is ​​output as the final sparse coding coefficients. The dimension of this final sparse coding coefficient matrix is ​​maintained at 1024 x 100, where the coding vector corresponding to each sample has a length of 1024. The number of non-zero elements contained is determined according to the final sparse constraint strength, and the proportion of non-zero elements is usually between 5% and 15%.

[0066] In one optional implementation, a sparse dictionary model is initialized, the spatiotemporal feature representation is used as input data, and the spatiotemporal feature representation is sparsely decomposed using the sparse dictionary model. Solving for the initial sparse coding coefficients under sparse constraints includes: Feature samples are extracted from the spatiotemporal domain feature representation, and clustering is performed on the feature samples. An initial set of dictionary atoms for the sparse dictionary model is constructed based on the cluster centers. An initial weight value is assigned to each dictionary atom in the initial set of dictionary atoms to complete the initialization of the sparse dictionary model. Using the spatiotemporal feature representation as input data, the spatiotemporal feature representation is linearly represented by the dictionary atom set in the sparse dictionary model, and the correlation between the spatiotemporal feature representation and the sparse coding coefficients is established. A sparse constraint is set, which limits the upper limit of the number of non-zero elements in the sparse coding coefficients. The sparse constraint is introduced into the sparse decomposition process as a solution constraint. Under the constraint of the sparse constraint, the initial sparse coding coefficients that satisfy the sparse constraint are solved by optimizing the error between the spatiotemporal domain feature representation and the reconstruction result of the sparse dictionary model.

[0067] Feature sample extraction is performed from the spatiotemporal feature representation. Assume the spatiotemporal feature representation is a three-dimensional tensor structure with dimensions of 128×64×32, corresponding to the time dimension, spatial row dimension, and spatial column dimension, respectively. A slice is selected along the time axis of this three-dimensional tensor every four time steps. Local feature blocks are extracted spatially using a sliding window approach, with a window size of 8×8 and a step size of 4. This extraction method yields 5000 feature sample vectors, each with a dimension of 64. These 5000 feature sample vectors are used as the input dataset for subsequent clustering processing.

[0068] K-means clustering is performed on the feature sample dataset, with a set number of clusters of 256, meaning 256 cluster centers need to be identified from the feature samples. 256 feature samples are randomly selected as initial cluster centers. The Euclidean distance between each feature sample and the 256 cluster centers is calculated, and each feature sample is assigned to the category of the nearest cluster center. The mean vector of all samples in each category is recalculated as the updated cluster center. This sample assignment and cluster center update operation is repeated until the change in cluster center position between two consecutive iterations is less than 0.001 or the number of iterations reaches 100. This yields 256 stable cluster center vectors, each with a dimension of 64.

[0069] Using the 256 cluster center vectors as the initial dictionary atoms for the sparse dictionary model, a dictionary matrix is ​​constructed. This matrix contains 256 columns, each corresponding to a dictionary atom, and each dictionary atom is a 64-dimensional vector. Normalization is performed on each dictionary atom in the dictionary matrix, and the L2 norm of each atom vector is calculated. Each element in the atom vector is divided by this L2 norm value to ensure that the L2 norm of each atom equals 1. A dictionary atom index table is created to record the numbers of the 256 dictionary atoms from 0 to 255.

[0070] Assign initial weights to each dictionary atom, creating a weight vector of length 256, where each element corresponds to a weight for a dictionary atom. Calculate the number of samples in each cluster of the feature sample clustering results, divide this number by the total number of feature samples (5000), and use the normalized frequency value as the initial weight for the corresponding dictionary atom. For example, if the first cluster contains 230 samples, the initial weight for the first dictionary atom is set to 0.046. Complete the initial weight assignment for all 256 dictionary atoms, ensuring the sum of all elements in the weight vector equals 1.

[0071] Using the spatiotemporal feature representation as input data for the sparse dictionary model, the three-dimensional tensor structure of the spatiotemporal feature representation needs to be converted into vector form. The 128×64×32 tensor is expanded into a one-dimensional vector with a length of 262144 in time-priority order. This one-dimensional vector is then segmented, with every 64 consecutive elements forming a feature vector segment, resulting in a total of 4096 feature vector segments. Each feature vector segment needs to be linearly represented using a dictionary set of atoms.

[0072] Sparse representation is solved for a single feature vector segment. The first feature vector segment, containing 64 numerical elements, is taken as the target to be represented. A linear combination of 256 dictionary atoms is used to construct a sparse coding coefficient vector of length 256. Each element in this sparse coding coefficient vector corresponds to the weight coefficient of a dictionary atom in the linear combination. A reconstructed vector, with a dimension of 64, is obtained by multiplying the dictionary matrix and the sparse coding coefficient vector, and is used to approximate the original feature vector segment.

[0073] A sparsity constraint is set to limit the number of non-zero elements in the sparse coding coefficient vector, stipulating that the number of non-zero elements in the sparse coding coefficient vector shall not exceed 12, that is, a maximum of 12 elements shall be selected from the 256 available dictionary atoms to participate in linear combination. This sparsity constraint ensures that the representation of the feature vector segment has high sparsity and selectivity, avoiding redundant representation caused by using too many dictionary atoms.

[0074] To solve for sparse coding coefficients under sparse constraints, an orthogonal matching pursuit algorithm is used for iterative solution. The residual vector is initialized as the feature vector segment to be represented, the sparse coding coefficient vector is initialized as an all-zero vector, and the selected dictionary atom set is initialized as an empty set. The inner product of the residual vector and 256 dictionary atoms is calculated, and the dictionary atom with the largest absolute value of the inner product is added to the selected dictionary atom set. Least-squares fitting is performed on the feature vector segment using the selected dictionary atom set to solve for the corresponding coding coefficient values, updating the elements at the corresponding positions in the sparse coding coefficient vector. The linear combination result of the selected dictionary atoms is calculated, and this linear combination result is subtracted from the feature vector segment to obtain a new residual vector. The dictionary atom selection, coefficient update, and residual calculation operations are repeated until the number of selected dictionary atoms reaches 12 or the L2 norm of the residual vector is less than 0.01.

[0075] A sparse coding coefficient vector satisfying the sparsity constraint is obtained. This vector has a length of 256, with at most 12 non-zero elements and the remaining 244 elements being 0. The index of the non-zero element corresponds to the dictionary atom number selected to participate in the linear representation, and the value of the non-zero element represents the contribution weight of the corresponding dictionary atom in the linear combination. For example, the elements at positions 5, 18, 47, 92, 105, 138, 177, 201, 223, and 245 of the sparse coding coefficient vector are 0.32, -0.18, 0.45, 0.27, -0.15, 0.38, -0.22, 0.19, 0.31, and -0.26, respectively, while the elements at all other positions are zero.

[0076] The above sparse representation solution process is performed on each of the 4096 feature vector segments to obtain 4096 sparse coding coefficient vectors. These 4096 sparse coding coefficient vectors are then organized into a two-dimensional matrix structure according to the original feature vector segment order. This matrix contains 4096 rows and 256 columns, with each row corresponding to the sparse coding coefficient of one feature vector segment. This two-dimensional matrix is ​​the initial sparse coding coefficient matrix for the spatiotemporal domain feature representation, completely recording the sparse decomposition results of the spatiotemporal domain feature representation under the sparse dictionary model.

[0077] In one alternative implementation, Feeding the reconstruction error back to the construction process of the adaptive sparse convolution kernel set, and updating the receptive field shape parameters and the spatial distribution density, includes: The density update and shape update are calculated based on the reconstruction error of the current iteration; when the reconstruction error is greater than a preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is increased and / or the receptive field major axis size is increased; when the reconstruction error is less than the preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is decreased and / or the receptive field major axis size is reduced. The major axis direction and / or major axis dimension in the receptive field shape parameters are updated based on the shape update amount, and boundary constraints are applied to the updated receptive field shape parameters. The adaptive sparse convolution kernel set is reconstructed using the updated spatial density and updated receptive field shape parameters, and then proceeds to the next iteration.

[0078] A second aspect of the present invention provides an asynchronous pulse convolution system for an integrated biomimetic visual sensing module, comprising: The acquisition module is used to acquire asynchronous event stream data output by the integrated bionic visual sensing module, perform spatiotemporal clustering analysis on the asynchronous event stream data, calculate local activity metrics based on the events in the asynchronous event stream data in the time and space dimensions, perform spatiotemporal clustering analysis on the asynchronous event stream data according to the local activity metrics, and divide the asynchronous event stream data into multiple event regions with different sparsity characteristics. The module is used to adaptively determine the spatial distribution density and receptive field shape parameters of sparse convolution kernels for each event region based on the corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. The convolution module is used to perform convolution operations on the asynchronous event stream data using the adaptive sparse convolution kernel set to extract spatiotemporal feature representations; iterative optimization is performed based on the spatiotemporal feature representations and the preset sparse dictionary model, and the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error in each iteration to obtain sparse coding coefficients; The closed-loop module is used to reconstruct the visual representation data of the target scene based on the sparse coding coefficients and the sparse dictionary model; feed the reconstruction error back to the construction process of the adaptive sparse convolution kernel set, update the receptive field shape parameters and the spatial distribution density, and form a closed-loop optimization.

[0079] A third aspect of the present invention provides an electronic device, comprising: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the aforementioned method.

[0080] A fourth aspect of the present invention provides a computer-readable storage medium having stored thereon computer program instructions that, when executed by a processor, implement the aforementioned method.

[0081] This invention can be a method, apparatus, system, and / or computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of the invention.

[0082] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. An asynchronous pulse convolution method for an integrated biomimetic visual sensing module that combines sensing, storage, and computation, characterized in that: include: Asynchronous event stream data output by the integrated bionic visual sensing module is acquired, spatiotemporal clustering analysis is performed on the asynchronous event stream data, local activity metrics are calculated based on the events in the asynchronous event stream data in the time and space dimensions, spatiotemporal clustering analysis is performed on the asynchronous event stream data according to the local activity metrics, and the asynchronous event stream data is divided into multiple event regions with different sparsity characteristics. For each event region, the spatial distribution density and receptive field shape parameters of the sparse convolution kernels are adaptively determined based on the corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. The asynchronous event stream data is subjected to convolution operations using the adaptive sparse convolution kernel set to extract spatiotemporal feature representations; Based on the spatiotemporal feature representation and the preset sparse dictionary model, iterative optimization is performed. In each iteration, the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error to obtain sparse coding coefficients. The visual representation data of the target scene is reconstructed based on the sparse coding coefficients and the sparse dictionary model; the reconstruction error is fed back to the construction process of the adaptive sparse convolution kernel set to update the receptive field shape parameters and the spatial distribution density, forming a closed-loop optimization.

2. The method according to claim 1, characterized in that, Based on the calculation of local activity metrics for events in asynchronous event stream data in both time and spatial dimensions, spatiotemporal clustering analysis is performed on the asynchronous event stream data according to the local activity metrics, dividing the asynchronous event stream data into multiple event regions with different sparsity characteristics, including: For each event in the asynchronous event stream data, extract its spatiotemporal coordinates, count the number of events within the spatial neighborhood within a preset time window, and obtain the time dimension event density and spatial dimension event density corresponding to each event. A spatiotemporal joint density function is constructed, and local activity metrics are calculated by interactively operating the event density in the time dimension and the event density in the spatial dimension based on the spatiotemporal joint density function. The local activity metrics corresponding to each event in the asynchronous event stream data are calculated based on the spatiotemporal joint density function to form a local activity metric distribution map. Using the local activity metric as a clustering feature, a clustering strategy based on density peak detection is adopted to group the asynchronous event stream data. The density peak points in the local activity metric distribution map are identified as cluster centers. Events with similar local activity metrics and continuous spatiotemporal domain are grouped into the same event region, forming multiple event regions with different sparsity characteristics.

3. The method according to claim 2, characterized in that, Constructing a spatiotemporal joint density function, and performing interactive calculations on the event density in the time dimension and the event density in the spatial dimension based on the spatiotemporal joint density function to obtain local activity metrics, including: Establish the basic expression form of the spatiotemporal joint density function, which includes a time dimension density term and a space dimension density term. The time dimension density term and the space dimension density term are obtained by substituting the time dimension event density and the space dimension event density into the time variable and the space variable of the spatiotemporal joint density function, respectively. The polarity information of each event in the asynchronous event stream data is extracted, and the polarity gradient feature is calculated based on the change pattern of the polarity information in the spatiotemporal domain. The polarity gradient feature is then used to modulate the weight coefficients of the time variables and spatial variables in the spatiotemporal joint density function. The modulated time-dimensional density term and the spatial-dimensional density term are used as inputs, and the time-dimensional density term and the spatial-dimensional density term are interactively operated on by a nonlinear coupling operator. The nonlinear coupling operator determines the coupling strength based on the mutual information between the time-dimensional density term and the spatial-dimensional density term. The local activity metric value corresponding to each event is calculated based on the operation result of the nonlinear coupling operator.

4. The method according to claim 1, characterized in that, For each event region, the spatial distribution density and receptive field shape parameters of the sparse convolution kernels are adaptively determined based on their corresponding local activity metrics, thereby constructing an adaptive set of sparse convolution kernels that matches the event distribution characteristics, including: Calculate the statistical distribution characteristics of the local activity metrics of all events within each event region, and substitute the statistical distribution characteristics into a preset density mapping function to establish a mapping relationship between the local activity metrics and the spatial distribution density of sparse convolution kernels; Based on the mapping relationship, the local activity metric is converted into a sparse convolution kernel spatial distribution density; The spatial diffusion degree of the event distribution is calculated based on the variance characteristics of the local activity metric, and principal component analysis is performed on the spatial distribution of the event using the spatial diffusion degree. The principal direction vector is extracted as the major axis direction in the receptive field shape parameter based on the results of the principal component analysis, and the major axis dimension in the receptive field shape parameter is determined based on the spatial diffusion degree. Based on the spatial distribution density of the sparse convolution kernels and the receptive field shape parameters, a corresponding number of sparse convolution kernels are generated within the event region; the sparse convolution kernels generated in all event regions are summarized to construct an adaptive sparse convolution kernel set that matches the event distribution characteristics.

5. The method according to claim 1, characterized in that, Based on the spatiotemporal feature representation and the preset sparse dictionary model, iterative optimization is performed. In each iteration, the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error to obtain sparse coding coefficients, including: Initialize the sparse dictionary model, take the spatiotemporal feature representation as input data, use the sparse dictionary model to perform sparse decomposition on the spatiotemporal feature representation, and solve the initial sparse coding coefficients under sparse constraints. Based on the initial sparse coding coefficients and the sparse dictionary model, reconstruct the spatiotemporal domain features, and calculate the reconstruction error between the reconstructed spatiotemporal domain features and the spatiotemporal domain feature representation; Based on the reconstruction error, calculate the sparse constraint strength adjustment and dictionary atom weight adjustment. Update the sparse constraint strength using the sparse constraint strength adjustment and update the weight value of each dictionary atom in the sparse dictionary model using the dictionary atom weight adjustment. Based on the updated sparse constraint strength and the updated sparse dictionary model, a new round of sparse decomposition is performed on the spatiotemporal feature representation to obtain updated sparse coding coefficients; a new reconstruction error is calculated based on the updated sparse coding coefficients, and it is determined whether the new reconstruction error meets the convergence condition. If the convergence condition is met, the current sparse coding coefficients are output.

6. The method according to claim 5, characterized in that, Initialize the sparse dictionary model, taking the spatiotemporal feature representation as input data, and perform sparse decomposition on the spatiotemporal feature representation using the sparse dictionary model. Solving for the initial sparse coding coefficients under sparse constraints includes: Feature samples are extracted from the spatiotemporal domain feature representation, and clustering is performed on the feature samples. An initial set of dictionary atoms for the sparse dictionary model is constructed based on the cluster centers. An initial weight value is assigned to each dictionary atom in the initial set of dictionary atoms to complete the initialization of the sparse dictionary model. Using the spatiotemporal feature representation as input data, the spatiotemporal feature representation is linearly represented by the dictionary atom set in the sparse dictionary model, and the correlation between the spatiotemporal feature representation and the sparse coding coefficients is established. A sparse constraint is set, which limits the upper limit of the number of non-zero elements in the sparse coding coefficients. The sparse constraint is introduced into the sparse decomposition process as a solution constraint. Under the constraint of the sparse constraint, the initial sparse coding coefficients that satisfy the sparse constraint are solved by optimizing the error between the spatiotemporal domain feature representation and the reconstruction result of the sparse dictionary model.

7. The method according to claim 1, characterized in that, Feeding the reconstruction error back to the construction process of the adaptive sparse convolution kernel set, and updating the receptive field shape parameters and the spatial distribution density, includes: The density update and shape update are calculated based on the reconstruction error of the current iteration; when the reconstruction error is greater than a preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is increased and / or the receptive field major axis size is increased; when the reconstruction error is less than the preset error threshold, the sparse convolution kernel spatial arrangement density of the corresponding event region is decreased and / or the receptive field major axis size is reduced. The major axis direction and / or major axis dimension in the receptive field shape parameters are updated based on the shape update amount, and boundary constraints are applied to the updated receptive field shape parameters. The adaptive sparse convolution kernel set is reconstructed using the updated spatial density and updated receptive field shape parameters, and then proceeds to the next iteration.

8. An asynchronous pulse convolution system for an integrated biomimetic visual sensing module, used to implement the method of any one of claims 1-7, characterized in that, include: The acquisition module is used to acquire asynchronous event stream data output by the integrated bionic visual sensing module, perform spatiotemporal clustering analysis on the asynchronous event stream data, calculate local activity metrics based on the events in the asynchronous event stream data in the time and space dimensions, perform spatiotemporal clustering analysis on the asynchronous event stream data according to the local activity metrics, and divide the asynchronous event stream data into multiple event regions with different sparsity characteristics. The module is used to adaptively determine the spatial distribution density and receptive field shape parameters of sparse convolution kernels for each event region based on the corresponding local activity metric, thereby constructing an adaptive sparse convolution kernel set that matches the event distribution characteristics. The convolution module is used to perform convolution operations on the asynchronous event stream data using the adaptive sparse convolution kernel set to extract spatiotemporal feature representations; Based on the spatiotemporal feature representation and the preset sparse dictionary model, iterative optimization is performed. In each iteration, the sparse constraint strength and dictionary atom weights are dynamically adjusted according to the reconstruction error to obtain sparse coding coefficients. A closed-loop module is used to reconstruct the visual representation data of the target scene based on the sparse coding coefficients and the sparse dictionary model. The reconstruction error is fed back into the construction process of the adaptive sparse convolution kernel set to update the receptive field shape parameters and the spatial distribution density, forming a closed-loop optimization.

9. An electronic device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to invoke instructions stored in the memory to execute the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images