A dynamic target detection method based on event camera adaptive pulse representation

Abstract

The application discloses a kind of dynamic target detection methods based on event camera adaptive pulse representation, belong to event vision, pulse neural network, target detection and intelligent early warning technical field.The method utilizes event camera to collect the asynchronous event stream generated by dynamic change in monitoring area, after region division, event preprocessing and adaptive pulse representation, complete dynamic target detection, risk level discrimination and early warning output.The application can improve the real-time, accuracy and robustness of target detection under complex illumination, high-speed motion, sudden intrusion and background disturbance scene, reduce the response lag and false alarm risk caused by fixed frame rate acquisition and fixed time window processing, suitable for public safety, traffic channel, industrial site, park perimeter and disaster monitoring scene.

Description

Technical Field

[0001] This invention belongs to the fields of event vision, spiking neural networks, target detection and intelligent early warning technology, and specifically relates to a dynamic target detection method based on adaptive pulse representation of an event camera. Background Technology

[0002] In scenarios such as transportation corridors, industrial park perimeters, industrial production, warehousing and logistics, construction sites, mine slopes, bridges and tunnels, urban roads, public safety, equipment operation, and disaster-prone areas, real-time detection and risk warning of dynamic objects such as moving targets, intruding targets, falling objects, sliding objects, abnormally moving objects, personnel, vehicles, equipment, or animals are typically required. These targets are characterized by uncertain appearance times, large fluctuations in movement speed, significant differences in target size, complex background environments, and high requirements for timely risk response. Failure to identify and assess risks in a timely manner can easily lead to equipment damage, road closures, personnel safety hazards, or production and operational shutdowns.

[0003] Existing dynamic target monitoring systems typically employ visible light cameras, infrared cameras, radar, lidar, vibration sensors, or multi-sensor fusion solutions of these devices. Among these, intelligent detection methods based on traditional video images rely on fixed frame rate sampling, requiring the transmission, storage, and computational analysis of continuous image frames. However, when most of the monitored scene is static or in a low-change state, these methods still generate a large amount of redundant data, further increasing the edge computing load and communication pressure. Furthermore, traditional video surveillance suffers significant image quality degradation in scenarios such as low illumination, strong light, shadows, day / night cycles, rain / fog, dust, motion blur, and sudden high-speed events, directly impacting the accuracy of dynamic target detection and low-latency early warning capabilities. While radar, lidar, or multi-sensor fusion solutions can improve some environmental adaptability, they generally suffer from high equipment costs, complex deployment and maintenance, limited spatial resolution, and difficulties in spatiotemporal synchronization of multi-source data.

[0004] However, the asynchronous event stream output by the event camera is not a regular image frame, but a sparse asynchronous sequence composed of spatial coordinates, timestamps, and event polarities. Existing methods typically convert the event stream into fixed-time-window event frames, fixed-number-of-event frames, voxel grids, or temporal surfaces before detection. However, the sampling granularity of these encoding methods is mostly manually set, making it difficult to adapt to changes in local event density and target motion speed, which easily leads to three problems: aliasing of details in high-speed target motion, response delay in sparse regions, and increased unnecessary computation.

[0005] Meanwhile, existing multi-scale feature fusion structures mostly follow the feature pyramid approach in ordinary image detection networks, which is mainly designed for continuous frame image features. For event stream pulse features, different levels of features not only contain spatial scale and semantic differences, but also pulse firing state, sparse activation distribution, and temporal response information. If ordinary convolutional feature fusion is directly used, the temporal firing information of the pulse features is easily weakened, thus affecting the detection stability of small-scale targets, fast-moving targets, and sudden intrusion targets. In addition, during the training of long-term event streams, spiking neural networks are also prone to problems such as unstable pulse firing rate, residual accumulation of membrane potential, and gradient fluctuations.

[0006] Therefore, there is an urgent need for an event stream dynamic target detection and early warning method that can achieve adaptive pulse sampling aggregation coding, pulse multi-scale feature fusion, and stable training. Summary of the Invention

[0007] In view of this, the present invention proposes a dynamic target detection method based on adaptive pulse representation of event camera to solve the problems existing in the above-mentioned event flow target detection methods, such as the difficulty in adapting to changes in local event density and target motion speed when using fixed time windows, fixed event number windows or ordinary framed event representations, resulting in overlapping details of high-speed target motion, delayed detection response in sparse regions, increased computational redundancy, and poor stability of long-term training of spiking neural networks.

[0008] The specific technical solution adopted in this invention is as follows:

[0009] A dynamic target detection method based on event camera adaptive pulse representation includes the following steps:

[0010] Step S1: Obtain the asynchronous event stream of the target monitoring area captured by the event camera, and divide the asynchronous event stream into multiple local event sequences according to the event coordinates and the preset monitoring area division rules; each event in the asynchronous event stream includes event coordinates, event occurrence time, and event polarity;

[0011] Step S2: Preprocess the multiple local event sequences to obtain the effective local event sequences corresponding to each sequence;

[0012] Step S3: For each effective local event sequence, use spiking neurons with memory characteristics to perform membrane potential integration, historical state preservation, and threshold firing judgment, and record the spiking time of each neuron; determine the adaptive sampling window based on the spiking times of two adjacent spiking times within the same local event sequence or the same spatial neighborhood.

[0013] Step S4: Within each adaptive sampling window, based on the spatiotemporal distribution characteristics of the event flow and the information transmission mechanism of the spiking neural network, multidimensional pulse state information that reflects local dynamic changes is extracted. Based on the multidimensional pulse state information, local event pulse representations corresponding to each sampling window are generated, and multiple local event pulse representations are organized into an event pulse feature map according to their corresponding spatial locations. The multidimensional pulse state information includes polymer film potential state, pulse firing state, event polarity, event time interval, local event density, and regional location encoding.

[0014] Step S5: Use a target detection network to extract multi-scale pulse features from the event pulse feature map to obtain pulse feature maps at different levels; fuse the pulse feature maps at different levels to generate multi-scale fused pulse features, and retain the sparse activation state and timing information of each level pulse feature map throughout the fusion process; input the multi-scale fused pulse features into the detection head for target prediction and output dynamic target detection results.

[0015] Furthermore, the method also includes a step of jointly training the spiking neurons, the target detection network, and the detection head before step S3;

[0016] The joint training employs a spatiotemporal adaptive training strategy and a residual membrane potential suppression strategy.

[0017] The spatiotemporal adaptive training strategy adaptively adjusts the pulse firing threshold, membrane potential forgetting coefficient, or local learning step size in an amplitude-controlled manner based on the pulse firing rate, membrane potential state, and threshold neighborhood state.

[0018] The residual membrane potential suppression strategy applies attenuation constraints, random inactivation constraints, or regularization constraints to the residual membrane potential after pulse firing or membrane potential reset.

[0019] Furthermore, in step S1, the target monitoring area is at least one of the following: traffic channels, park perimeter, industrial production area, warehousing and logistics area, construction site, mine slope, bridge and tunnel, urban road, public safety area, equipment operation area, water conservancy facility perimeter, or disaster-prone area; the asynchronous event stream is triggered by one or more of the following: moving target, intruding target, falling object, sliding object, abnormally moving object, personnel, vehicle, equipment, animal, or environmental disturbance.

[0020] Furthermore, in step S1, the rules for dividing the monitoring area include:

[0021] Based on the fixed structural boundaries, warning lines, or safety boundaries within the target monitoring area, the target monitoring area is divided into multiple local monitoring areas;

[0022] Generate regional location codes based on the geometric extent of each local monitoring area;

[0023] Based on the historical event density distribution or target occurrence priors corresponding to each local monitoring area, the initial attention level of each local monitoring area is determined, and regional risk weights are generated based on the initial attention level.

[0024] The event coordinates in the asynchronous event stream are mapped to the corresponding local monitoring areas to form a local event sequence with regional location encoding and regional risk weight.

[0025] Furthermore, in step S2, the event preprocessing includes at least one of isolated noise filtering, flicker interference suppression, background activity filtering, periodic disturbance suppression, non-target event screening, or non-critical area event deweighting.

[0026] Furthermore, in step S3, the adaptive sampling window differs from the fixed time window, fixed event number window, or ordinary frame accumulation method. It is generated by triggering the sampling window method when the membrane potential of the spiking neuron reaches or exceeds the firing threshold.

[0027] The membrane potential integral is updated using an accumulation method with a forgetting mechanism. The specific update formula is as follows:

[0028] ;

[0029] in, For the first A spiking neuron in the event The membrane potential after arrival; The membrane potential before the event occurs; This is the membrane potential forgetting coefficient; The stimulus response at the current input time is determined by the polarity of the current input event, the time interval between the current event and the previous event, and the spatial location of the current event. For the first Learnable parameters of each spiking neuron;

[0030] The threshold issuance determination is performed in the following manner:

[0031] ;

[0032] in, In pulse delivery state, It is a step function. For the first The firing threshold of a single spiking neuron;

[0033] The adaptive sampling window is defined by the times of two adjacent pulse firings, and is expressed as follows:

[0034] ;

[0035] in, Indicates the first At the time of the kth impulse firing of a neuron This represents the corresponding adaptive sampling window.

[0036] Furthermore, in step S4, within each adaptive sampling window, the polymer film potential state, pulse firing state, event polarity, event time interval, local event density, and region location encoding are used to generate the calculation formula for the local event pulse representation corresponding to each sampling window.

[0037] ;

[0038] in To represent the local event impulses in the r-th local monitoring area within the k-th adaptive sampling window, For aggregate functions, This is the membrane potential state. In pulse delivery state, For the polarity of the event, This indicates the regional location code or regional risk weight. An adaptive sampling window;

[0039] The local event pulse representations of each local monitoring area are rearranged according to their spatial location to form an event pulse feature map; the event pulse feature map includes at least one of the following: membrane potential channel, pulse emission channel, positive polarity event channel, negative polarity event channel, time interval channel, and regional location encoding channel.

[0040] Furthermore, the target detection network includes a pulse feature extraction module, a pulse feature pyramid fusion module, and a detection head module;

[0041] The pulse feature extraction module is composed of a pulse convolutional layer and at least one of the following modules stacked together: a pulse residual module, a pulse dense connection module, and a pulse attention module. Its input is an event pulse feature map, which is used to extract pulse feature maps of different levels from the event pulse feature map. During the feature extraction process, the pulse feature extraction module uses the membrane potential state of the pulse neurons at each level as one of the inputs to the next level, so that the pulse firing state, sparse activation distribution, and temporal response information are passed down level by level. The different levels include shallow feature maps that represent local motion details and edge responses, mid-level feature maps that represent target region information, and high-level feature maps that represent deep semantic information.

[0042] The pulse feature pyramid fusion module receives pulse feature maps of different levels output by the pulse feature extraction module. It performs channel adjustment and scale alignment on the pulse feature maps output at each level, and then fuses the adjusted pulse feature maps at each level to generate multi-scale fused pulse features. The channel adjustment refers to adjusting the number of channels of each level feature map to a uniform value using pointwise convolution. The scale alignment refers to adjusting the spatial size of each level feature map to a uniform size through upsampling or downsampling. The fusion involves adding the feature maps at each level after channel adjustment and scale alignment element by element, concatenating them, and then performing convolutional dimensionality reduction or weighted averaging.

[0043] The detection head module receives multi-scale fused pulse features output by the pulse feature pyramid fusion module, and performs target classification and bounding box regression on the multi-scale fused pulse features to output target detection results; the target detection results include at least one of target category, target location, target bounding box, target motion trend and detection confidence.

[0044] Furthermore, the dynamic target detection method based on event camera adaptive pulse representation further includes step S6, analyzing the target state based on the dynamic target detection results, specifically including:

[0045] S61. Obtain the target detection results for each frame within the continuous sampling window, and extract the target's category, location, bounding box, motion direction, motion state, and detection confidence.

[0046] S62. Generate the target's temporal trajectory information based on the extracted information;

[0047] S63. When a target is stably detected within multiple consecutive sampling windows, it is determined to be a continuously valid target; when the position, bounding box, or direction of movement of the target changes, the target state is dynamically updated; when the detection confidence of the target is lower than a preset threshold or the target does not appear continuously within the consecutive sampling windows, it is determined to be an invalid target or a transient noise response.

[0048] S64. Output target state information, wherein the target state information includes at least one of the following: target category, target location, target bounding box, motion direction, motion state, detection confidence, and corresponding time information.

[0049] By adopting the above technical solution, the present invention has at least the following beneficial effects:

[0050] 1. This invention uses the asynchronous event stream output by the event camera as the input for dynamic target detection. Events are generated only when the pixel brightness changes to the trigger condition. Compared with the traditional fixed frame rate video acquisition method, it can reduce redundant data caused by static background areas and improve the response speed to moving targets, intruding targets, falling objects, sliding objects and abnormally moving objects.

[0051] 2. This invention proposes an adaptive pulse sampling aggregation coding method driven by the integral of the membrane potential of spiking neurons and the threshold firing state, which is different from fixed time window, fixed event number window or ordinary frame accumulation method, so that the event representation can be dynamically adjusted with the local event density and target motion intensity.

[0052] 3. This invention determines the adaptive sampling window by the timing of adjacent pulse emission, so that the event-dense region or high-speed moving target corresponds to a shorter sampling window to retain rapidly changing information; and the event-sparse region or low-speed changing region corresponds to a longer sampling window to reduce invalid sampling and redundant calculation, thereby balancing the real-time detection and the completeness of event representation.

[0053] 4. This invention performs joint aggregation of membrane potential evolution state, pulse firing state, event polarity, event time interval, local event density, and regional location encoding within an adaptive sampling window to form an event pulse feature map for dynamic target detection tasks. Compared with simple event frames, voxel grids, or event histograms, it is better able to express the temporal triggering characteristics and motion change features of dynamic targets.

[0054] 5. This invention performs scale alignment, channel adjustment, and cross-scale fusion on pulse feature maps of different levels through pulse feature pyramid fusion, enabling the detection process to simultaneously utilize edge, motion, and local detail information in shallow pulse features, as well as semantic information and target region information in deep pulse features, thereby improving the detection performance of small-scale targets, fast-moving targets, sudden intrusion targets, and low-contrast dynamic targets.

[0055] 6. In the process of multi-scale fusion, the present invention retains the sparse activation state and temporal delivery information of pulse features, reduces the loss of temporal information between event encoding and detection network, and enables the pulse temporal features obtained by front-end adaptive pulse sampling aggregation encoding to continue to be transmitted in the subsequent detection stage.

[0056] 7. This invention introduces a spatiotemporal adaptive training strategy and a residual membrane potential suppression strategy. By constraining the pulse firing rate, membrane potential state, threshold neighborhood state, and residual membrane potential, it alleviates problems such as excessively strong pulse firing, excessively weak pulse firing, residual membrane potential accumulation, and training oscillation, thereby improving the stability of the target detection training process and the reliability of the detection results in long-term event streams. Attached Figure Description

[0057] Figure 1 This is an overall flowchart of the dynamic target detection and early warning method of the present invention;

[0058] Figure 2 This is a flowchart of the event stream acquisition, region division, and preprocessing process of the present invention;

[0059] Figure 3 This is a flowchart of the adaptive pulse sampling aggregation coding process of the present invention;

[0060] Figure 4 This is a schematic diagram of the pulse feature extraction module in the embodiment;

[0061] Figure 5 This is a flowchart of the pulse feature extraction, multi-scale fusion, and target detection process of the present invention. Detailed Implementation

[0062] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the following embodiments are only used to illustrate the technical solution of the present invention and do not constitute a limitation on the scope of protection of the present invention. Without departing from the concept of the present invention, those skilled in the art can make appropriate adjustments to the monitoring scene, region division method, spiking neuron parameters, sampling window triggering conditions, feature fusion method, training constraints, and early warning threshold according to actual needs.

[0063] like Figure 1 As shown in the figure, this embodiment provides a dynamic target detection method based on adaptive pulse representation of event cameras, including the following steps: event stream acquisition and monitoring area division, event preprocessing, adaptive pulse sampling and sampling window generation, construction of event pulse feature map, and target detection using a target detection network. The steps of event stream acquisition and monitoring area division and event preprocessing are described in [reference needed]. Figure 2 For the steps of adaptive pulse sampling, sampling window generation, and construction of event pulse feature maps, please refer to [link / reference]. Figure 3 For reference on object detection using object detection networks, see [link / reference]. Figure 4 The detailed execution process for each step is as follows:

[0064] Step S1, Event Stream Acquisition and Monitoring Area Division:

[0065] S11, deploy an event camera within the target monitoring area, such that its field of view covers at least the key prevention sub-area within the target monitoring area; the key prevention sub-area includes at least one of a warning area and a prohibited intrusion area, wherein the warning area is located outside the prohibited intrusion area and the outer boundary of the warning area is a security boundary; the field of view of the event camera also covers the background area surrounding the key prevention sub-area, wherein the background area is used to provide a background reference for moving target detection.

[0066] The target monitoring area includes at least one of the following: traffic channels, park perimeter, industrial production area, warehousing and logistics area, construction site, mine slope, bridge and tunnel, urban road, public safety area, equipment operation area, water conservancy facility perimeter, and disaster-prone area; the event camera is suitable for low light, strong light, backlight, shadow changes, day and night alternation, rain and fog, dust, or high-speed movement scenes.

[0067] S12, acquire the asynchronous event stream collected by the event camera. The asynchronous event stream is generated by one or more of the following triggers: moving target, intruding target, falling object, sliding object, abnormally moving object, person, vehicle, equipment, animal, or environmental disturbance. A single event is represented as:

[0068] ;

[0069] in, and The spatial coordinates representing the occurrence of the event. Indicates the time when the event occurred. Indicates the polarity of the event.

[0070] S13, according to the preset monitoring area division rules, the field of view of the event camera is divided into multiple local monitoring areas, and each event in the asynchronous event stream is mapped to the corresponding local monitoring area according to the event coordinates to form multiple local event sequences;

[0071] The rules for dividing the monitoring area include:

[0072] Based on the fixed structural boundaries, warning lines, or security boundaries within the target monitoring area, the target monitoring area is divided into multiple local monitoring areas; the local monitoring areas are either regular grid areas or irregular areas divided according to the actual scene structure, business rules, or security boundaries.

[0073] Generate regional location codes based on the geometric extent of each local monitoring area;

[0074] Based on the historical event density distribution or target occurrence frequency corresponding to each local monitoring area, the initial attention weight of each local monitoring area is determined, and the regional spatial sampling weight is generated according to the initial attention weight. Specifically, higher regional attention weights are assigned to areas where targets frequently appear or where event activity is dense, while lower regional attention weights are assigned to areas with less event activity. This embodiment uses a relative ranking method instead of absolute value setting. The number of events generated within a unit of time within the area is ranked in real time, and the frequency is determined by the percentile ranking of that value across all areas. Specifically, the attention weight of the monitored area is divided into a continuous interval [0.1, 2.0].

[0075] When the number of regional events ranks in the top 10% of all events, it is given the highest rank weight, i.e., >1.5, and is considered extremely frequent.

[0076] When the ranking is between the top 10% and top 25%, a weight of 0.8-1.2 is assigned to the mid-to-high level, which is considered to be relatively frequent.

[0077] When a player ranks in the bottom 25%, only a minimum weight of 0.1-0.3 is assigned, which is considered a low-frequency region.

[0078] This dynamic percentile mapping ensures that the threshold for "frequent" events adapts and drifts with changes in the overall environment without human intervention.

[0079] The event sequence within the r-th local monitoring region can be represented as:

[0080] ;

[0081] in, Indicates the first The coordinate range of a local monitoring area.

[0082] The region location encoding and region spatial sampling weights are used for subsequent adaptive pulse sampling, event pulse feature construction, and multi-scale target detection processes.

[0083] Step S2: Preprocess the multiple local event sequences respectively, selecting at least one of the following preprocessing operations to obtain the valid local event sequences corresponding to each sequence:

[0084] Isolated noise filtering: For the current event, a preset spatial and temporal neighborhood is defined centered on its spatial coordinates and timestamp, and the number of neighboring events within this spatiotemporal range is counted; if the number does not meet the preset neighbor count requirement, the event is considered to lack sufficient spatiotemporal context support, is judged as isolated noise, and is deleted.

[0085] Flicker interference suppression: For each pixel location, monitor the temporal density of event triggering. If a pixel exhibits an explosive triggering that far exceeds the normal activity level within a short window, and the burst does not have the continuous trajectory characteristics of a moving target, it is identified as an interference event caused by light source flicker, and all events generated by that pixel within that time period are deleted.

[0086] Background activity filtering: For each pixel location, its long-term historical event rate is statistically analyzed as the background baseline. If the event triggering intensity of a certain pixel in the current time period is consistently below the baseline and does not show the polarity change pattern unique to the target event, it is determined to be irrelevant background activity and its event is deleted from the sequence.

[0087] Periodic disturbance suppression: For each pixel position, detect whether there is a fixed periodic rhythm in the event triggering, and at the same time examine whether the distribution ratio of positive and negative polarity events tends to be balanced and whether the spatial position remains stable within this period; if all three conditions are met, it is determined that it is caused by equipment vibration or mechanical periodic disturbance, and all events of that pixel within this period are deleted.

[0088] Non-target event filtering: Based on the preset rules for determining the effective spatial area, only events whose spatial location falls within the effective area are retained, while events outside the area are discarded.

[0089] Non-critical region event weight reduction processing: According to the preset regional importance classification rules, events in different spatial locations are assigned different weight coefficients, and events in non-critical regions have lower weights than those in critical regions; this operation only adjusts the contribution of events in subsequent membrane potential integration, and does not delete the events themselves.

[0090] It should be noted that the above operations are independent of each other, and one or more can be flexibly selected according to the actual scenario during implementation; when multiple operations are selected, deletion operations are executed first, and demotion operations are executed last. After all preprocessing is completed, the remaining events are then treated as a valid local event sequence.

[0091] Step S3, Adaptive Pulse Sampling and Window Generation:

[0092] This embodiment uses a spiking neuron model with memory properties to perform temporal integration processing on each effective local event sequence. Specifically:

[0093] Each neuron maintains a time-decreasing membrane potential variable to characterize the cumulative activity level of events within its local region. Whenever a new event arrives, the membrane potential integral is updated using an accumulation method with a forgetting mechanism, based on the event polarity, arrival time interval, and local event density. The membrane potential integral update method is expressed mathematically as follows:

[0094] ;

[0095] in, For the first A spiking neuron in the event The membrane potential after arrival; The membrane potential before the event occurs; This is the membrane potential forgetting coefficient; The stimulus response at the current input time is determined by the polarity of the current input event, the time interval between the current event and the previous event, and the spatial location of the current event. For the first Learnable parameters of each spiking neuron;

[0096] When the accumulated membrane potential of any neuron reaches or exceeds a preset firing threshold, the neuron immediately generates a pulse output and records the current time as the pulse firing time of that neuron. After the pulse is generated, the membrane potential undergoes a reset operation (subtracting the threshold or clearing it to zero) to begin the next round of integration. The threshold firing state is expressed by the following mathematical formula:

[0097] ;

[0098] in, In pulse delivery state, It is a step function. For the first The firing threshold of a single spiking neuron;

[0099] Based on the recorded pulse firing times of each neuron, the boundaries of the adaptive sampling window are determined: for any local event sequence, the midpoint between two adjacent pulse firing times of neurons within the sequence is used as the window dividing point, dividing the continuous event stream into sampling segments of unequal length; for multiple spatially adjacent local sequences, a joint sampling interval is defined by the earliest and latest pulse firing times in these sequences. Accordingly, the adaptive sampling window, defined by two adjacent pulse firing times, is expressed as:

[0100] ;

[0101] in, Indicates the first At the time of the kth impulse firing of a neuron This represents the corresponding adaptive sampling window.

[0102] The adaptive sampling window in this embodiment differs from existing window methods based on fixed rule segmentation. The fixed rules include segmentation by fixed duration, segmentation by fixed number of events, or accumulation into frames by fixed period. The window position and width are driven by the dynamic response of the spiking neuron itself, thereby achieving automatic following of changes in information density in the event stream.

[0103] Step S4: Within each adaptive sampling window, based on the asynchronous temporal characteristics of the event flow and the spatiotemporal dynamic information expression requirements of the spiking neural network, extract multidimensional pulse state information to describe local event changes, including membrane potential state, pulse firing state, event polarity, event time interval, local event density, and region location encoding. Among these, membrane potential state is used to characterize the temporal response intensity during event accumulation, pulse firing state is used to describe the discrete activation information generated by neuron response, event polarity is used to indicate the direction of brightness change, event time interval is used to reflect the speed of local motion change, local event density is used to describe the intensity of event activity within the target area, and region location encoding is used to retain the spatial location information of the event occurrence.

[0104] The following formula is used for calculation:

[0105] ;

[0106] in To represent the local event impulses in the r-th local monitoring area within the k-th adaptive sampling window, For aggregate functions, This is the membrane potential state. In pulse delivery state, For the polarity of the event, This indicates the regional location code or regional risk weight. This is an adaptive sampling window.

[0107] The local event pulse features generated in each local monitoring area are rearranged according to their spatial position to form an event pulse feature map; the event pulse feature map includes at least one of the following: membrane potential feature channel, pulse emission feature channel, positive polarity event statistical feature channel, negative polarity event statistical feature channel, time interval feature channel, local event density feature channel, and regional location encoding channel.

[0108] Step S5: Use an object detection network for prediction. The object detection network used in this embodiment is as follows: Figure 5 As shown, the system includes a pulse feature extraction module, a pulse feature pyramid fusion module, and a detection head module. This allows for better utilization of temporal context information in subsequent target detection, significantly improving the robustness of identifying fast-moving, occluded, and blurred targets. In this embodiment, the event pulse feature maps corresponding to the continuously adaptive sampling window are further combined into a temporal feature sequence to serve as input data for the target detection network.

[0109] like Figure 4As shown, the pulse feature extraction module includes a backbone network and feature output terminals connected to each layer. The backbone network, along the signal transmission direction, includes: a Stem layer, layer 1 (shallow layer), layer 2 (middle layer), and layer 3 (high layer). The Stem layer is configured to receive the raw input signal and encode it into an initial pulse sequence or membrane potential state, and its output terminal is communicatively connected to the input terminal of layer 1.

[0110] Layer 1 contains, in series, a pulsed convolution module, a pulsed residual module, and a pulsed attention module. The pulsed convolution module serves as the input of Layer 1, receiving signals from the Stem layer; the pulsed attention module serves as the output of Layer 1, generating and outputting shallow features p1, and also connecting to the input of Layer 2.

[0111] Layers 2 and 3 have the same structure, and both add a pulsed dense connection module between the pulsed residual module and the pulsed attention module. Unlike layer 1, the signals from layers 2 and 3 sequentially pass through the pulsed convolution module, the pulsed residual module, and the pulsed dense connection module before being connected to the pulsed attention module. Specifically, the output of the pulsed attention module in layer 2 generates the mid-level feature pm and connects to the input of layer 3; the output of the pulsed attention module in layer 3 generates the high-level feature pL and connects to the membrane potential state transfer module. The membrane potential state transfer module is used to perform subsequent state update or reset operations to maintain the network's temporal characteristics.

[0112] The pulse feature extraction module outputs pulse feature maps at different levels, including shallow pulse feature maps, mid-level pulse feature maps, and high-level pulse feature maps. The shallow pulse feature map is used to characterize the local spatial structure information in the input event pulse, the mid-level pulse feature map is used to characterize the structural features of the target region, and the high-level pulse feature map is used to characterize the semantic features of the target.

[0113] The pulse feature pyramid fusion module receives pulse feature maps of different levels output by the pulse feature extraction module, and performs channel adjustment and scale alignment processing on each level of pulse feature map. The channel adjustment adopts a pointwise convolution operation to adjust the number of channels of pulse feature maps of different levels to a uniform dimension. The scale alignment adopts an upsampling or downsampling operation to adjust the spatial size of pulse feature maps of different levels to a uniform scale. Subsequently, the pulse feature maps of different levels after channel adjustment and scale alignment are fused to generate multi-scale fused pulse features.

[0114] The multi-scale fusion process is implemented by element-wise addition, feature convolution followed by dimensionality reduction, weighted fusion, or a combination of the above methods. The process can be represented as follows:

[0115] ;

[0116] in, This represents the generated multi-scale fused pulse features. This represents different levels of pulse feature maps output by the pulse feature extraction module. This represents a multi-scale feature fusion function.

[0117] The detection head module receives multi-scale fused pulse features output by the pulse feature pyramid fusion module, performs target classification and bounding box regression on the fused pulse features, and outputs dynamic target detection results.

[0118] In one implementation, the output of the detection head module can be expressed as:

[0119] ;

[0120] Where Y represents the dynamic target detection result, This represents the detection head mapping function. This represents the multi-scale fused pulse characteristics.

[0121] Step S6, analyze the target state based on the dynamic target detection results, specifically including:

[0122] S61. Obtain the target detection results for each frame within the continuous sampling window, and extract the target's category, location, bounding box, motion direction, motion state, and detection confidence.

[0123] S62. Generate the target's temporal trajectory information based on the extracted information;

[0124] S63. When a target is stably detected within multiple consecutive sampling windows, it is determined to be a continuously valid target; when the position, bounding box, or direction of movement of the target changes, the target state is dynamically updated; when the detection confidence of the target is lower than a preset threshold or the target does not appear continuously within the consecutive sampling windows, it is determined to be an invalid target or a transient noise response.

[0125] S64. Output target state information, wherein the target state information includes at least one of the following: target category, target location, target bounding box, motion direction, motion state, detection confidence, and corresponding time information.

[0126] In the above method, the spiking neuron model and the target detection network are trained using a joint training method, the specific steps of which are as follows:

[0127] The joint training includes a spatiotemporal adaptive training strategy and a residual membrane potential suppression strategy;

[0128] The spatiotemporal adaptive training strategy adaptively adjusts the pulse firing threshold, membrane potential forgetting coefficient, or local learning parameters based on the pulse firing state, membrane potential change state, and threshold neighborhood response state during training. When the pulse firing frequency is lower than a preset range, the neuron's response capability is improved by lowering the firing threshold, reducing the membrane potential decay coefficient, or enhancing the input response. When the pulse firing frequency is higher than a preset range, abnormal firing is restricted by increasing the firing threshold, increasing the membrane potential decay coefficient, or reducing the local learning parameters, so that the pulse response remains stable.

[0129] The residual membrane potential suppression strategy is used to reduce the accumulation of residual membrane potential in spiking neurons during continuous event input. It applies at least one of the following constraints to the membrane potential state after pulse firing: attenuation constraint, random inactivation constraint, or regularization constraint, in order to reduce the state accumulation error under continuous time steps.

[0130] The model training objective is composed of target detection loss, residual membrane potential constraint loss, and pulse firing stability constraint loss.

[0131] ;

[0132] in, This represents the total training loss of the model. Indicates the target detection loss. This represents the residual membrane potential confinement loss. This indicates the pulse firing stability constraint loss. and This represents the corresponding loss weight coefficient.

Claims

1. A dynamic target detection method based on event camera adaptive pulse representation, characterized in that, Includes the following steps: Step S1: Obtain the asynchronous event stream of the target monitoring area captured by the event camera, and divide the asynchronous event stream into multiple local event sequences according to the event coordinates and the preset monitoring area division rules; each event in the asynchronous event stream includes event coordinates, event occurrence time, and event polarity; Step S2: Preprocess the multiple local event sequences to obtain the effective local event sequences corresponding to each sequence; Step S3: For each effective local event sequence, use spiking neurons with memory characteristics to perform membrane potential integration, historical state preservation, and threshold firing judgment, and record the spiking time of each neuron; determine the adaptive sampling window based on the spiking times of two adjacent spiking times within the same local event sequence or the same spatial neighborhood. Step S4: Within each adaptive sampling window, based on the spatiotemporal distribution characteristics of the event flow and the information transmission mechanism of the spiking neural network, extract multidimensional pulse state information that can reflect local dynamic changes, generate local event pulse representations corresponding to each sampling window based on the multidimensional pulse state information, and organize multiple local event pulse representations into an event pulse feature map according to their corresponding spatial locations. The multidimensional pulse state information includes polymer film potential state, pulse firing state, event polarity, event time interval, local event density, and regional location encoding; Step S5: Use an object detection network to extract multi-scale pulse features from the event pulse feature map to obtain pulse feature maps at different levels; Multi-scale fused pulse features are generated by fusing pulse feature maps of different levels, and the sparse activation state and timing information of each level pulse feature map are preserved throughout the fusion process. Multi-scale fused pulse features are input into the detection head for target prediction, and dynamic target detection results are output. The dynamic target detection results include target category, target location, target bounding box, target motion trend, and detection confidence.

2. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, The method further includes a step of jointly training the spiking neurons, the target detection network, and the detection head before step S3; The joint training employs a spatiotemporal adaptive training strategy and a residual membrane potential suppression strategy. The spatiotemporal adaptive training strategy adaptively adjusts the pulse firing threshold, membrane potential forgetting coefficient, or local learning step size in an amplitude-controlled manner based on the pulse firing rate, membrane potential state, and threshold neighborhood state. The residual membrane potential suppression strategy applies attenuation constraints, random inactivation constraints, or regularization constraints to the residual membrane potential after pulse firing or membrane potential reset.

3. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, In step S1, the target monitoring area is at least one of the following: traffic channels, park perimeter, industrial production area, warehousing and logistics area, construction site, mine slope, bridge and tunnel, urban road, public safety area, equipment operation area, water conservancy facility perimeter, or disaster-prone area; the asynchronous event stream is triggered by one or more of the following: moving target, intruding target, falling object, sliding object, abnormally moving object, personnel, vehicle, equipment, animal, or environmental disturbance.

4. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, In step S1, the monitoring area division rules include: Based on the fixed structural boundaries, warning lines, or safety boundaries within the target monitoring area, the target monitoring area is divided into multiple local monitoring areas; Generate regional location codes based on the geometric extent of each local monitoring area; Based on the historical event density distribution or target occurrence priors corresponding to each local monitoring area, the initial attention level of each local monitoring area is determined, and regional risk weights are generated based on the initial attention level. The event coordinates in the asynchronous event stream are mapped to the corresponding local monitoring areas to form a local event sequence with regional location encoding and regional risk weight.

5. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, In step S2, the event preprocessing includes at least one of isolated noise filtering, flicker interference suppression, background activity filtering, periodic disturbance suppression, non-target event screening, or non-critical area event deweighting.

6. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, In step S3, the adaptive sampling window is different from the fixed time window, fixed event number window or ordinary frame accumulation method. It is generated by the sampling window method triggered when the membrane potential of the spiking neuron reaches or exceeds the firing threshold. The membrane potential integral is updated using an accumulation method with a forgetting mechanism. The specific update formula is as follows: ； in, For the first A spiking neuron in the event The membrane potential after arrival; The membrane potential before the event occurs; This is the membrane potential forgetting coefficient; The stimulus response at the current input time is determined by the polarity of the current input event, the time interval between the current event and the previous event, and the spatial location of the current event. For the first Learnable parameters of each spiking neuron; The threshold issuance determination is performed in the following manner: ； in, In pulse delivery state, It is a step function. For the first The firing threshold of a single spiking neuron; The adaptive sampling window is defined by the times of two adjacent pulse firings, and is expressed as follows: ； in, Indicates the first At the time of the kth impulse firing of a neuron This represents the corresponding adaptive sampling window.

7. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, In step S4, within each adaptive sampling window, the polymer film potential state, pulse firing state, event polarity, event time interval, local event density, and region location encoding are used to generate the calculation formula for the local event pulse representation corresponding to each sampling window. ； in To represent the local event impulses in the r-th local monitoring area within the k-th adaptive sampling window, It is an aggregate function. This is the membrane potential state. In pulse delivery state, For the polarity of the event, This indicates the regional location code or regional risk weight. For adaptive sampling window; The local event pulse representations of each local monitoring area are rearranged according to their spatial location to form an event pulse feature map; The event pulse feature map includes at least one of the following: membrane potential channel, pulse emission channel, positive polarity event channel, negative polarity event channel, time interval channel, and regional location encoding channel.

8. The dynamic target detection method based on event camera adaptive pulse representation according to claim 1, characterized in that, The target detection network includes a pulse feature extraction module, a pulse feature pyramid fusion module, and a detection head module; The pulse feature extraction module is composed of a pulse convolutional layer and at least one of the following modules stacked together: a pulse residual module, a pulse dense connection module, and a pulse attention module. Its input is an event pulse feature map, which is used to extract pulse feature maps of different levels from the event pulse feature map. In the feature extraction process, the pulse feature extraction module uses the membrane potential state of the pulse neurons at each level as one of the inputs to the next level, so that the pulse firing state, sparse activation distribution, and temporal response information are passed down level by level. The pulse feature pyramid fusion module receives pulse feature maps of different levels output by the pulse feature extraction module. It performs channel adjustment and scale alignment on the pulse feature maps output at each level, and then fuses the adjusted pulse feature maps at each level to generate multi-scale fused pulse features. The channel adjustment refers to adjusting the number of channels of each level feature map to a uniform value using pointwise convolution. The scale alignment refers to adjusting the spatial size of each level feature map to a uniform size through upsampling or downsampling. The fusion involves adding the feature maps at each level after channel adjustment and scale alignment element by element, concatenating them, and then performing convolutional dimensionality reduction or weighted averaging. The detection head module receives multi-scale fused pulse features output by the pulse feature pyramid fusion module, and performs target classification and bounding box regression on the multi-scale fused pulse features to output target detection results; the target detection results include at least one of target category, target location, target bounding box, target motion trend and detection confidence.

9. A dynamic target detection method based on event camera adaptive pulse representation according to any one of claims 1 to 8, characterized in that, The method further includes step S6, analyzing the target state based on the dynamic target detection results, specifically including: S61. Obtain the target detection results for each frame within the continuous sampling window, and extract the target's category, location, bounding box, motion direction, motion state, and detection confidence. S62. Generate the target's temporal trajectory information based on the extracted information; S63. When a target is stably detected within multiple consecutive sampling windows, it is determined to be a continuously valid target; when the position, bounding box, or direction of movement of the target changes, the target state is dynamically updated; when the detection confidence of the target is lower than a preset threshold or the target does not appear continuously within the consecutive sampling windows, it is determined to be an invalid target or a transient noise response. S64. Output target state information, wherein the target state information includes at least one of the following: target category, target location, target bounding box, motion direction, motion state, detection confidence, and corresponding time information.

Images