Multi-target video intelligent detection method based on artificial intelligence and related equipment

Abstract

The application relates to the technical field of computer vision, and provides a multi-target video intelligent detection method based on artificial intelligence and related equipment, which comprises the following steps: organizing event signals in a video signal into a sparse event stream sequence in time sequence; grouping and aggregating the event signals in the sparse event stream sequence to form event blocks; extracting local space-time features of the event blocks to obtain event tensors; inputting the event tensors into a pulse neural network model to obtain a pulse distribution; the pulse neural network model is provided with a predictive potential accumulation mechanism and a local plasticity adjustment mechanism; the predictive potential accumulation mechanism adjusts a response coefficient of an input pulse signal based on a membrane potential change state of a target historical moment of a neuron; the local plasticity adjustment mechanism adjusts a transmission strength of a synapse connection based on a time difference of arrival of the input pulse signal and an output pulse signal; and a target detection result is generated based on the pulse distribution. The technical scheme provided by the application improves target detection accuracy.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a multi-target video intelligent detection method and related equipment based on artificial intelligence. Background Technology

[0002] With the rapid development of artificial intelligence and computer vision technologies, video object detection methods based on convolutional neural networks have been widely applied in fields such as intelligent surveillance, autonomous driving, and industrial inspection. Traditional video detection systems typically acquire continuous image frames at a fixed frame rate and perform feature extraction and object recognition operations on each frame. These methods can achieve high detection accuracy in static or low-dynamic scenes, but in scenes with high-speed motion, changing lighting, or dense targets, frame-based sampling generates a large amount of redundant data, leading to increased information processing latency and system power consumption.

[0003] In recent years, the emergence of event camera technology has provided new insights into high dynamic range visual perception. Event cameras trigger event output by detecting changes in pixel brightness, featuring high temporal resolution, low latency, and sparse output. However, most existing event-driven detection methods still rely on traditional neural network structures, failing to adequately utilize the temporal distribution characteristics and sparsity of event signals. They also lack the ability to predict the changing trends of input events, and synaptic weight updates depend on global error signals, making real-time adaptive adjustment difficult and resulting in poor target detection accuracy. Summary of the Invention

[0004] This invention provides a multi-target video intelligent detection method and related equipment based on artificial intelligence, in order to overcome the deficiencies in the prior art.

[0005] This invention provides an artificial intelligence-based intelligent multi-target video detection method, comprising: Acquire video signals and organize the event signals in the video signals into a sparse event stream sequence according to time order; Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks; Extract the local spatiotemporal features of the event block to obtain the event tensor; The event tensor is input into a spiking neural network model to obtain a pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity regulation mechanism. The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment. The local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal. The target event region is obtained by performing spatiotemporal mapping on the pulse distribution, and the target detection result is generated based on the feature representation of the target event region.

[0006] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein the predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment, including: A change state recording unit is established for each neuron in the spiking neural network model, and the change state recording unit stores the direction and magnitude of the increase or decrease of the membrane potential of the neuron at the previous moment; When performing membrane potential updates, the response coefficients corresponding to the current input pulse signals of each neuron are adjusted according to the direction of increase or decrease of the membrane potential; wherein, if the membrane potential is determined to be increasing based on the direction of increase or decrease of the membrane potential, the response coefficient is increased, and if the membrane potential is determined to be decreasing based on the direction of increase or decrease of the membrane potential, the response coefficient is decreased.

[0007] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein the local plasticity adjustment mechanism adjusts the transmission strength of synaptic connections based on the arrival time difference between input pulse signals and output pulse signals, including: Obtain the arrival time difference between the input pulse signal and the output pulse signal in the synaptic connection; The transmission strength of the synaptic connection is dynamically adjusted based on the arrival time difference, wherein the transmission strength is increased when the arrival time of the input pulse signal is earlier than the arrival time of the output pulse signal, and the transmission strength is decreased when the arrival time of the input pulse signal is later than the arrival time of the output pulse signal. Based on the adjusted transmission intensity, the pulse distribution output by the spiking neural network model is updated synchronously.

[0008] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein generating target detection results based on the feature representation of the target event region includes: The number of event pulses per unit time is counted, and the number of event pulses is compared with a preset wake-up threshold. If the number of event pulses is greater than or equal to the wake-up threshold, the detection network in a dormant state is woken up; Based on the feature representation of the target event region, the detection network generates the target detection result, which includes the target category, confidence level, and spatial coordinates.

[0009] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein the length of the preset time window is adjusted based on the event triggering frequency; wherein, when the event triggering frequency increases, the length of the preset time window is shortened, and when the event triggering frequency decreases, the length of the preset time window is extended.

[0010] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, the method further includes: The number of times the event signal is triggered is counted according to a preset time period, and the event triggering frequency is calculated based on the number of triggers. If the event trigger frequency is greater than or equal to a preset trigger rate threshold, the signal threshold for generating the event signal is increased. If the event triggering frequency is less than the preset triggering rate threshold, the signal threshold for generating the event signal is reduced.

[0011] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein the step of performing spatiotemporal mapping on the pulse distribution to obtain a target event region, and generating a target detection result based on the feature representation of the target event region, includes: The pulse distribution is mapped in time and space to determine the active event region that is continuously activated within a continuous time slice as the target event region; A connectivity analysis is performed on the target event region, and based on the connectivity analysis results, the target event region is divided into multiple candidate target regions; According to spatial and temporal order, the event signals in each candidate target region are superimposed to generate a region-level feature representation; Based on the region-level feature representation, the target detection result containing target category, confidence level, and spatial coordinates is generated.

[0012] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein the spiking neural network model includes an input encoding layer, a pulse propagation layer, and a pulse decoding layer; the step of inputting the event tensor into the spiking neural network model to obtain the pulse distribution includes: Based on the input encoding layer, the time-series information in the event tensor is transformed into a pulse sequence; wherein the pulse sequence preserves the temporal order of the event signals; Based on the pulse sequence, the output pulse signal is obtained by performing accumulation and firing operations on the temporal relationship of the input pulse signal through each neuron in the pulse propagation layer. Based on the pulse decoding layer, the output pulse signal is time-series aggregated to generate the pulse distribution.

[0013] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein organizing the event signals in the video signal into a sparse event stream sequence according to time order includes: The video signal is subjected to noise reduction and brightness equalization processing; In adjacent video frames of the processed video signal, if the brightness change of the same pixel is detected to be greater than or equal to a preset brightness change threshold, an event signal containing pixel coordinates, brightness change polarity, and time information is generated. All generated event signals are arranged in chronological order to construct the sparse event stream sequence.

[0014] According to the present invention, a multi-object video intelligent detection method based on artificial intelligence is provided, wherein extracting the local spatiotemporal features of the event block to obtain the event tensor includes: Within each event block, the direction of event change is determined according to the time sequence of the event signals, and a spatiotemporal correspondence is established based on the time interval and spatial position difference between adjacent event signals. Based on the spatiotemporal correspondence, the event change trend information is statistically analyzed, and the event change trend information is encoded according to the time dimension and the spatial dimension to generate the local spatiotemporal features; The local spatiotemporal features are formatted to generate the event tensor.

[0015] According to the present invention, a multi-target video intelligent detection method based on artificial intelligence is provided, wherein the step of performing spatiotemporal mapping on the pulse distribution to obtain a target event region, and generating a target detection result based on the feature representation of the target event region, includes: Based on the target detection results of multiple consecutive video signals, candidate matching pairs are established. Calculate the consistency index of each candidate matching pair in terms of spatial location and pulse frequency; Based on the consistency index, the matching relationship of the target objects in the video signal is determined; Based on the matching relationship, a trajectory sequence for each target object is generated.

[0016] The present invention also provides a multi-target video intelligent detection device based on artificial intelligence, comprising: The acquisition module is configured to acquire video signals and organize the event signals in the video signals into a sparse event stream sequence according to the time sequence. The aggregation module is configured to group and aggregate the event signals in the sparse event stream sequence based on a preset time window and spatial location association to form event blocks. The extraction module is configured to extract the local spatiotemporal features of the event block to obtain the event tensor; The input module is configured to input the event tensor into a spiking neural network model to obtain a pulse distribution; wherein, the spiking neural network model is provided with a predictive potential accumulation mechanism and a local plasticity regulation mechanism, the predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment, and the local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal; The detection module is configured to perform spatiotemporal mapping on the pulse distribution to obtain the target event region, and generate a target detection result based on the feature representation of the target event region.

[0017] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the artificial intelligence-based multi-target video intelligent detection method as described above.

[0018] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the artificial intelligence-based multi-target video intelligent detection method as described above.

[0019] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the artificial intelligence-based multi-target video intelligent detection method as described above.

[0020] This invention provides an AI-based multi-target video intelligent detection method and related equipment. By extracting event signals from video signals and constructing a sparse event stream sequence, it eliminates a large amount of static background redundancy data generated by traditional video frame-based sampling at the data source, significantly reducing the latency and power consumption of subsequent information processing. Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks, structuring highly discrete asynchronous event points and initially reconstructing the local spatiotemporal contour of the moving target. The local spatiotemporal features of the event blocks are extracted to obtain the event tensor. The event tensor is input into a spiking neural network model to obtain the pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity adjustment mechanism. The predictive potential accumulation mechanism endows neurons with the ability to remember and predict. Referring to the membrane potential changes of neurons at historical moments of the target, the response coefficient to the current input pulse signal is dynamically adjusted, enabling neurons to adaptively adjust their sensitivity according to the changing rhythm of the event stream. The local plasticity adjustment mechanism, by monitoring the time difference between the two ends of a synaptic connection (i.e., the input pulse signal and the output pulse signal), directly adjusts the transmission strength of the synaptic connection in real time locally, achieving weight adaptation consistent with the biological principle of time-related plasticity. The technical solution provided by this invention significantly improves the model's response sensitivity and multi-target detection accuracy in high-dynamic, high-speed motion, or sudden illumination changes. By inversely mapping the high-dimensional pulse distribution back to actual spatiotemporal physical coordinates, the target event region is delineated. Finally, the target detection result is obtained based on the feature representations within these target event regions. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in this invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0022] Figure 1 This is a flowchart illustrating the multi-target video intelligent detection method based on artificial intelligence provided by the present invention.

[0023] Figure 2 This is a schematic diagram of the structure of the multi-target video intelligent detection device based on artificial intelligence provided by the present invention.

[0024] Figure 3 This is a schematic diagram of the structure of the electronic device provided by the present invention. Detailed Implementation

[0025] To make the objectives, technical solutions, and advantages of this invention clearer, the technical solutions of this invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of this invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without creative effort are within the scope of protection of this invention.

[0026] The data involved in this application (including but not limited to data used for analysis, data stored, data displayed, etc.) are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0027] Figure 1 This is a flowchart illustrating an artificial intelligence-based intelligent multi-target video detection method according to an exemplary embodiment. Figure 1 As shown in an exemplary embodiment, the multi-target video intelligent detection method based on artificial intelligence includes steps 110 to 150, which are described in detail below: Step 110: Acquire video signals and organize the event signals in the video signals into a sparse event stream sequence according to time order.

[0028] In this embodiment of the invention, the video signal is a visual data stream containing continuous temporal states acquired from an image acquisition device (such as a vision sensor, event camera, etc.). This video signal can cover visual information ranging from static or low-dynamic scenes to high-speed motion and scenes with drastic changes in lighting.

[0029] Event signals are asynchronous signals triggered by changes in pixel brightness. That is, when a target in the scene moves or the lighting changes, only pixels with changes in brightness will output event signals, while areas such as the background that have not changed will not output them.

[0030] Since event signals are generated only when there are dynamic changes, the sequence exhibits high sparsity in both the time and spatial domains. A sparse event stream sequence is a data sequence formed by arranging the aforementioned discrete, asynchronously triggered event signals in the order in which they were triggered (i.e., time sequence).

[0031] By extracting event signals and constructing a sparse event stream sequence, a large amount of static background redundant data generated by traditional video frame sampling is eliminated from the data source, which can significantly reduce the latency and power consumption of subsequent information processing.

[0032] Step 120: Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks.

[0033] In this embodiment of the invention, a small time slice (e.g., 10 milliseconds or 30 milliseconds) is captured on the timeline as a preset time window. The preset time window determines the span of events that can be identified within one processing cycle.

[0034] Spatial location correlation refers to the distance or topological relationship between different event signals on a two-dimensional pixel plane, such as spatial proximity and temporal continuity. Within a preset time window, event signals that are spatially adjacent and temporally continuous are grouped together to form event blocks. Specifically, after receiving a sparse event stream sequence, a spatiotemporal joint approach is used to group event signals within the same preset time window and meeting a certain distance threshold in spatial coordinates (such as adjacent pixels or a defined neighborhood) into the same group to form event blocks. This structures the highly discrete asynchronous event points and initially reconstructs the local spatiotemporal contour of the moving target.

[0035] In this embodiment of the invention, after the event signals are generated, they are grouped according to a preset time window. Within each time window, the event signals are sorted in chronological order, and the proximity of adjacent event signals in spatial location is determined sequentially. When adjacent event signals are continuous in time and their spatial location is within a preset proximity range, they are merged into the same event block. When the above conditions are not met, a new event block is started to be recorded.

[0036] Step 130: Extract the local spatiotemporal features of the event block to obtain the event tensor.

[0037] In this embodiment of the invention, local spatiotemporal features of the event block are extracted. Local spatiotemporal features are feature expressions that describe the changing patterns of event signals within the event block in the time and space dimensions, such as the brightness change gradient, motion direction, and motion speed in the local area.

[0038] Since spiking neural network models typically require structured data of fixed dimensions as input, the event tensor is obtained by formatting, gridding, or tensorizing the local spatiotemporal features. The event tensor then converts unstructured event blocks into standard inputs for the spiking neural network model.

[0039] Step 140: Input the event tensor into the spiking neural network model to obtain the pulse distribution; wherein, the spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity regulation mechanism. The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment. The local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal.

[0040] In this embodiment of the invention, the Spiking Neural Network (SNN) model is a neural network inspired by the biological brain, in which neurons communicate through discrete pulses, making it suitable for processing event tensors with high temporal resolution.

[0041] In this embodiment of the invention, the spiking neural network model adopts a leaky integral firing neuron model. Simultaneously, the spiking neural network model introduces a predictive potential accumulation mechanism and a local plasticity regulation mechanism, which are described below: In the predictive potential accumulation mechanism, it no longer passively accumulates potentials at a fixed pace, but dynamically adjusts the response coefficient to the current input pulse signal by referring to the membrane potential change state of the neuron at the target historical moment (such as the previous moment or several historical moments), so as to endow the neuron with the ability to remember and predict, and enable the neuron to adaptively adjust its sensitivity according to the changing rhythm of the event flow. The local plasticity regulation mechanism monitors the arrival time difference between the two ends of the synaptic connection (i.e., the input pulse signal and the output pulse signal) and directly adjusts the transmission strength of the synaptic connection in real time locally, thereby achieving weight adaptation that conforms to the biological time-related plasticity principle.

[0042] The pulse distribution is the set of activation states of the output pulse signal in the spatial and temporal dimensions after the spiking neural network model has processed the input event tensor.

[0043] Step 150: Spatiotemporal mapping of the pulse distribution is performed to obtain the target event region, and target detection results are generated based on the feature representation of the target event region.

[0044] In this embodiment of the invention, the target event region is delineated by inversely mapping the high-dimensional pulse distribution back to the actual spatiotemporal physical coordinates. The target event region is a connected region with high-density activation in the pulse distribution, which typically corresponds to a moving target that actually exists in the scene. Finally, based on the feature representations within these target event regions, operations such as classification and bounding box regression are performed to generate target detection results containing information such as target type and location coordinates.

[0045] In this embodiment of the invention, by extracting event signals from video signals and constructing a sparse event stream sequence, a large amount of static background redundant data generated by traditional video frame-based sampling is eliminated from the data source, significantly reducing the latency and power consumption of subsequent information processing. Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks, structuring highly discrete asynchronous event points and initially reconstructing the local spatiotemporal contour of the moving target. The local spatiotemporal features of the event blocks are extracted to obtain the event tensor. The event tensor is input into a spiking neural network model to obtain the pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity adjustment mechanism. The predictive potential accumulation mechanism endows neurons with the ability to remember and predict, dynamically adjusting the response coefficient to the current input pulse signal by referring to the membrane potential change state of the neuron at the target's historical moment, enabling the neuron to adaptively adjust its sensitivity according to the changing rhythm of the event stream. The local plasticity adjustment mechanism, by monitoring the time difference between the two ends of the synaptic connection (i.e., the input pulse signal and the output pulse signal), directly adjusts the transmission strength of the synaptic connection in real time locally, realizing weight adaptation in accordance with the biological principle of time-related plasticity. The technical solution provided by this invention significantly improves the model's response sensitivity and multi-target detection accuracy in high-dynamic, high-speed motion, or sudden illumination changes. By inversely mapping the high-dimensional pulse distribution back to actual spatiotemporal physical coordinates, target event regions are delineated. Finally, target detection results are obtained based on the feature representations within these target event regions.

[0046] In an exemplary embodiment of the present invention, the predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state at a target historical moment of the neuron, including: A change state recording unit is established for each neuron in the spiking neural network model, and the change state recording unit stores the direction and magnitude of the increase or decrease of the membrane potential of the neuron at the previous moment; When performing membrane potential updates, the response coefficients corresponding to the current input pulse signals of each neuron are adjusted according to the direction of increase or decrease of the membrane potential; wherein, if the membrane potential is determined to be increasing based on the direction of increase or decrease of the membrane potential, the response coefficient is increased, and if the membrane potential is determined to be decreasing based on the direction of increase or decrease of the membrane potential, the response coefficient is decreased.

[0047] In this embodiment of the invention, a change-state recording unit is established for each neuron in the spiking neural network model. This unit stores the direction and magnitude of the increase / decrease in the neuron's membrane potential at the previous time step, functioning as a short-term memory component for the neuron. During membrane potential updates, the response coefficient corresponding to the current input pulse signal for each neuron is adjusted based on the direction of membrane potential increase / decrease. At the next time step, the neuron references the change-state record from the previous time step when calculating the current membrane potential. Specifically, if the membrane potential shows an upward trend based on the direction of increase / decrease, the response coefficient to the input pulse signal at this time step is increased, allowing the neuron to accumulate potential faster when input events increase. If the membrane potential shows a downward trend based on the direction of increase / decrease, the response coefficient is decreased accordingly, slowing down the rate of potential increase caused by the input pulse signal. If the record indicates that the membrane potential remains stable, the response coefficient remains unchanged. When several consecutive time steps show an upward trend in membrane potential, the neuron increases the accumulated weight of the input pulse signal before reaching the firing threshold. When a sudden increase in input events is detected, the neuron can reach the firing condition in a shorter time, thus achieving early firing.

[0048] This adjustment of the response coefficient is maintained over several consecutive update cycles, enabling neurons to fire in advance when the frequency of event changes increases. Once in a steady state, the averaged response coefficient is maintained until a significant trend in membrane potential change is detected again. Through this predictive potential accumulation mechanism, neurons develop a forward-looking response to sudden events, ensuring the network maintains real-time detection even when event frequency suddenly increases.

[0049] In video signals, the presence of high-speed moving targets causes a sharp increase in the trigger frequency of corresponding event signals. At this time, the membrane potential of the corresponding neuron in the spiking neural network model will inevitably show a continuous upward trend from the previous moment. Upon sensing this upward trend, the neuron will actively increase its response coefficient to the current input pulse signal (e.g., by increasing the cumulative step size). This allows the neuron's membrane potential to break through the firing threshold more quickly, forming an early response firing pattern, thus significantly shortening the detection delay for high-speed moving targets. Conversely, if the signal is in a stable or interference-reducing period (membrane potential decreases), the response coefficient will decrease to suppress false triggering due to noise.

[0050] In an exemplary embodiment of the present invention, the local plasticity adjustment mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal, including: Obtain the arrival time difference between the input pulse signal and the output pulse signal in the synaptic connection; The transmission strength of the synaptic connection is dynamically adjusted based on the arrival time difference, wherein the transmission strength is increased when the arrival time of the input pulse signal is earlier than the arrival time of the output pulse signal, and the transmission strength is decreased when the arrival time of the input pulse signal is later than the arrival time of the output pulse signal. Based on the adjusted transmission intensity, the pulse distribution output by the spiking neural network model is updated synchronously.

[0051] In this embodiment of the invention, each synaptic connection includes a time difference comparison unit and a connection adjustment unit. The time difference comparison unit is used to detect the arrival time difference between the input pulse signal and the output pulse signal. That is, during the operation of the spiking neural network model, the pulse emission time of each synaptic connection's input and output ends is recorded respectively. After the current time step is completed, the difference between the arrival time of the input pulse signal and the emission time of the corresponding output pulse signal is calculated to obtain the arrival time difference. When the input pulse signal is earlier than the output pulse signal in time, the arrival time difference is determined to be a positive time difference; when the input pulse is later than the output pulse, the arrival time difference is determined to be a negative time difference; when the two occur almost simultaneously, the time difference is determined to be zero.

[0052] The connection adjustment unit adjusts the transmission strength of the synaptic connections based on the arrival time difference; that is, it updates the transmission strength of each synaptic connection according to the arrival time difference. Specifically, when the arrival time difference is positive, it indicates that the input pulse signal has a forward promoting effect on the output pulse signal, increasing the transmission strength of that synaptic connection; when the arrival time difference is negative, it indicates that the input pulse signal lags behind the output pulse signal, weakening the transmission strength of that synaptic connection; when the arrival time difference is close to zero, the synaptic strength remains unchanged.

[0053] The aforementioned transmission strength adjustment is performed based on the synaptic state at the current time step, and the adjustment range is limited by a preset maximum change step size to prevent excessive fluctuations in synaptic strength. After each adjustment, the firing records of the synaptic connection in the following time steps are continuously tracked. If a positive time difference trend is maintained for several consecutive time steps, the transmission strength of the synapse is continuously increased; if a negative time difference trend is maintained for several consecutive time steps, the transmission strength of the synapse is continuously decreased. When positive and negative trends alternate, the transmission strength of the synaptic connection is smoothly updated to maintain the overall stability of the network connection. After adjustment, the pulse distribution characteristics output by the pulse propagation layer in the spiking neural network model are updated synchronously.

[0054] The local plasticity adjustment mechanism enables spiking neural network models to utilize the temporal asymmetry of the sparse event flow itself to perform real-time, adaptive parameter fine-tuning during inference.

[0055] In an exemplary embodiment of the present invention, generating a target detection result based on the feature representation of the target event region includes: The number of event pulses per unit time is counted, and the number of event pulses is compared with a preset wake-up threshold. If the number of event pulses is greater than or equal to the wake-up threshold, the detection network in a dormant state is woken up; Based on the feature representation of the target event region, the detection network generates the target detection result, which includes the target category, confidence level, and spatial coordinates.

[0056] In this embodiment of the invention, the number of event pulses per unit time is counted and compared with a preset wake-up threshold. The unit time can be a fixed time interval, such as 10 milliseconds or 100 milliseconds, or a time unit related to the video processing frame rate. The number of event pulses refers to the total number of pulses output by the spiking neural network model within the aforementioned unit time, which directly reflects the dynamic activity level of the current scene. The wake-up threshold is a preset value used to distinguish whether the scene is in a static, low-dynamic, or high-dynamic state. This threshold can be set according to the application scenario or system power consumption budget.

[0057] If the number of event pulses is greater than or equal to the wake-up threshold, it indicates that there are significant dynamic targets or high-density motion information in the current scene. At this time, the detection network in a dormant state is awakened, and then based on the feature representation of the target event region, the detection network generates a target detection result containing target category, confidence level, and spatial coordinates. Conversely, if the number of event pulses is less than the wake-up threshold, it indicates that the current scene is a static background, and the detection network remains dormant.

[0058] Detection networks are network modules used to perform complex object classification and precise localization, such as a lightweight convolutional neural network or other classifiers.

[0059] In this embodiment of the invention, the high-power detection network can remain dormant for most of the time when there is no target or the target is stationary, and the front-end monitoring is performed only by the ultra-low-power spiking neural network model. The detection network is only woken up when the spiking neural network model detects sufficient dynamic information, thereby reducing average power consumption and unnecessary computational latency.

[0060] In an exemplary embodiment of the present invention, the length of the preset time window is adjusted based on the event triggering frequency; wherein, when the event triggering frequency increases, the length of the preset time window is shortened, and when the event triggering frequency decreases, the length of the preset time window is extended.

[0061] In this embodiment of the invention, the event triggering frequency is the total number of event signals generated per unit time, which is a direct indicator of the intensity of dynamic changes in the scene.

[0062] The length of the preset time window is dynamically adjusted based on the event trigger frequency. The adjustment rules are as follows: when the event trigger frequency increases compared to the previous time window, it means the target is moving quickly or the scene is changing drastically. To capture more detailed instantaneous dynamics, the length of the preset time window needs to be shortened. Multiple preset step sizes can be set for the preset time window length. When shortening is needed, the preset time window length is reduced by one step. When the event trigger frequency decreases compared to the previous time window, it means the scene is relatively stable. The preset time window length can be appropriately extended to aggregate more complete slow-motion information and reduce the number of subsequent processing batches. The preset time window length is extended by one step. The adjustment of the preset time window length remains between the preset minimum and maximum boundaries and takes effect at the start of the next time window.

[0063] In this embodiment of the invention, by dynamically linking the length of the time window with the event triggering frequency, the granularity of data aggregation is adaptively adjusted. In high-dynamic scenarios, it automatically switches to a high frame rate mode to capture details with smaller time slices; in low-dynamic scenarios, it automatically switches to a low frame rate mode to integrate information with coarser granularity. This makes the formation of event blocks more intelligent, ensuring that key details are not lost during high-speed motion and avoiding the generation of too much fragmented data during low-speed motion, thereby optimizing processing efficiency and resource utilization under different dynamic conditions.

[0064] In an exemplary embodiment of the present invention, organizing the event signals in the video signal into a sparse event stream sequence according to time order includes: The video signal is subjected to noise reduction and brightness equalization processing; In adjacent video frames of the processed video signal, if the brightness change of the same pixel is detected to be greater than or equal to a preset brightness change threshold, an event signal containing pixel coordinates, brightness change polarity, and time information is generated. All generated event signals are arranged in chronological order to construct the sparse event stream sequence.

[0065] In this embodiment of the invention, the video signal includes a color video signal or a grayscale video signal. For the color video signal, after extracting its luminance component, the same difference and threshold judgment process as for the grayscale video signal is performed. Specifically, before performing luminance difference calculation, the video signal undergoes denoising and luminance equalization processing. The denoising process uses a median filtering scheme, and the luminance equalization process uses local histogram adjustment.

[0066] Pixel-level brightness difference calculation is performed on consecutive video frames. The difference result is compared with a preset brightness change threshold. When the brightness change of the same pixel in an adjacent video frame is greater than or equal to the preset brightness change threshold, it is determined that the pixel has undergone a significant brightness change. An event signal is then generated, and the corresponding pixel coordinates, brightness change polarity, and event occurrence time information are recorded. The brightness change polarity is determined by judging the sign of the brightness difference between the current video frame and the previous video frame, and is used to distinguish between two directions of brightness change: rising and falling. That is, when the brightness change of the same pixel in an adjacent video frame is greater than or equal to the preset brightness change threshold, an event signal containing pixel coordinates, brightness change polarity, and time information is generated.

[0067] The detected event signals are organized into a sparse event stream sequence in chronological order and queued in the event buffer according to time priority, laying the data foundation for accurate inference of subsequent models.

[0068] In one exemplary embodiment of the present invention, the method further includes: The number of times the event signal is triggered is counted according to a preset time period, and the event triggering frequency is calculated based on the number of triggers. If the event trigger frequency is greater than or equal to a preset trigger rate threshold, the signal threshold for generating the event signal is increased. If the event triggering frequency is less than the preset triggering rate threshold, the signal threshold for generating the event signal is reduced.

[0069] In this embodiment of the invention, the number of times an event signal is triggered is counted according to a preset time period, the event trigger frequency is calculated based on the number of triggers, and the calculated event trigger frequency is compared with a preset trigger rate threshold, which defines the maximum data processing load that can be tolerated.

[0070] If the event trigger frequency is greater than or equal to the preset trigger rate threshold, it indicates frequent event triggering and increased power consumption, which may lead to system overheating or excessive energy consumption. In this case, the signal threshold for generating event signals needs to be increased, i.e., the preset brightness change threshold needs to be increased. Increasing the preset brightness change threshold means that only more drastic brightness changes can generate event signals, thus throttling data flow. If the event trigger frequency is less than the preset trigger rate threshold, it indicates insufficient event triggering and decreased detection sensitivity. In this case, the preset brightness change threshold needs to be decreased to improve sensitivity to subtle dynamic changes.

[0071] When it is determined that the preset brightness change threshold needs to be adjusted, the preset brightness change threshold is modified step by step within the preset adjustment step size range; when the event trigger frequency is higher than the upper limit, the preset brightness change threshold is increased proportionally to improve the event signal generation conditions; when the event trigger frequency is lower than the lower limit, the preset brightness change threshold is decreased proportionally to enhance the event signal generation sensitivity.

[0072] The updated preset brightness change threshold is loaded into the event signal generation stage, and the event signal acquisition and detection process is re-executed within the new time period. The change in event trigger frequency before and after the adjustment is recorded. When the event trigger frequency tends to stabilize within several consecutive periods, the current preset brightness change threshold is kept unchanged.

[0073] The technical solution provided by the embodiments of the present invention can automatically reduce the preset brightness change threshold in extreme scenarios such as sudden changes in lighting or dense targets to avoid data flooding and system overload; while when the scene is stable, it can automatically increase the preset brightness change threshold to capture potential minute details.

[0074] In an exemplary embodiment of the present invention, the step of performing spatiotemporal mapping on the pulse distribution to obtain a target event region, and generating a target detection result based on the feature representation of the target event region, includes: The pulse distribution is mapped in time and space to determine the active event region that is continuously activated within a continuous time slice as the target event region; A connectivity analysis is performed on the target event region, and based on the connectivity analysis results, the target event region is divided into multiple candidate target regions; According to spatial and temporal order, the event signals in each candidate target region are superimposed to generate a region-level feature representation; Based on the region-level feature representation, the target detection result containing target category, confidence level, and spatial coordinates is generated.

[0075] In this embodiment of the invention, the pulse distribution output by the spiking neural network model is mapped temporally and spatially to determine the event regions that are continuously active within consecutive time slices. When the pulse density at a certain spatial location is consistently higher than the average level within a consecutive time slice, that location is marked as an active event pixel. Between adjacent time slices, the active event pixels are compared. When the positions of active event pixels in multiple consecutive time slices are adjacent or overlap, they are merged into the same active event region, which is then used as the target event region.

[0076] Connectivity analysis is performed on the target event region. For example, a connected component labeling algorithm is used for connectivity analysis. When a group of pixels meets the eight-neighbor connectivity condition in space, they are divided into the same candidate target region. If the minimum boundary distance between active event regions exceeds a preset distance threshold, they are regarded as different candidate target regions.

[0077] For each candidate target region, the event signals are superimposed according to spatial and temporal order, and the spatial coordinates, timestamps and pulse density values ​​of each event are extracted. The summation is then performed in the time dimension to generate a region-level feature representation.

[0078] Based on the regional feature representation, target category determination and location estimation are performed. According to the density, duration and spatial shape of the event activity in the region, the similarity is compared with the preset target feature template. When the similarity exceeds the threshold, the target category corresponding to the candidate target region is determined. At the same time, the bounding rectangle boundary of the candidate target region is calculated, and its center coordinates are used as the target spatial location. The confidence level is estimated based on the intensity of the pulse activity.

[0079] After identifying candidate target regions, the number of pulses emitted within the current time window is counted, and the average pulse intensity is calculated based on the number of pulses emitted. When the average pulse intensity exceeds a preset activity intensity threshold, and the intensity variation in a continuous time slice is lower than a stability threshold, the candidate target region is determined to have a stable response, and the confidence level is estimated to be high. When the average pulse intensity does not reach the activity intensity threshold, or the variation in a continuous time slice exceeds the stability threshold, the candidate target region is determined to have an unstable response, and the confidence level is estimated to be low. Finally, the confidence level is converted into a numerical output according to the corresponding ratio.

[0080] The identification results of all candidate target regions are organized according to their confidence level and time sequence to form a set of detection results.

[0081] In an exemplary embodiment of the present invention, the spiking neural network model includes an input encoding layer, a pulse propagation layer, and a pulse decoding layer; the step of inputting the event tensor into the spiking neural network model to obtain the pulse distribution includes: Based on the input encoding layer, the time-series information in the event tensor is transformed into a pulse sequence; wherein the pulse sequence preserves the temporal order of the event signals; Based on the pulse sequence, the output pulse signal is obtained by performing accumulation and firing operations on the temporal relationship of the input pulse signal through each neuron in the pulse propagation layer. Based on the pulse decoding layer, the output pulse signal is time-series aggregated to generate the pulse distribution.

[0082] In this embodiment of the invention, the spiking neural network model includes an input encoding layer, a pulse propagation layer, and a pulse decoding layer.

[0083] The input encoding layer transforms the time-series information in the event tensor into a pulse sequence while preserving the temporal order of the event signals. Specifically, during the input encoding stage, the time index and spatial coordinate information corresponding to each event signal in the event tensor are mapped to a discrete-time sequence. Based on the temporal order of the event signals, an independent time step number is assigned to each event signal, and the positive and negative directions of the pulses are set according to the event polarity. During the encoding process, the original temporal order of the event signals is maintained, ensuring that the encoded pulse sequence reflects the true temporal distribution of the event signals.

[0084] In the pulse propagation layer, leak-integral firing neurons carrying a predictive potential accumulation mechanism perform accumulation and firing operations according to the temporal relationship of the input pulse signals. That is, during the pulse propagation phase, each leak-integral firing neuron gradually accumulates the input pulse signal. When the accumulated membrane potential exceeds a set firing threshold, the neuron fires an output pulse signal at the corresponding time step. If the firing threshold is not reached within consecutive time steps, the membrane potential gradually decays at the leakage rate. The accumulation and firing process is repeated at each time step, thus forming a sparse yet ordered pulse firing sequence in the time dimension.

[0085] The pulse propagation layer employs an improved Leaky Integrate-and-Fire (LIF) neuron, which introduces a predictive potential accumulation term into the membrane potential update formula based on a predictive potential accumulation mechanism, while also incorporating a time-difference-based synaptic plasticity mechanism.

[0086] The pulse decoding layer performs temporal aggregation of the output pulse signals from multiple neurons to generate a pulse distribution characterizing the overall event activity intensity. Specifically, after the pulse propagation layer outputs the pulse firing sequence, it summarizes the firing activities of different neurons at the same time step, calculates the number of pulses and their distribution density per unit time, and performs a weighted average of the firing results from multiple neurons to obtain a pulse density value representing the overall event activity level. The change curve of the pulse density value is recorded over a continuous time window, forming a pulse distribution characterizing the temporal features of the overall event, which can be used for subsequent connectivity analysis and target location estimation.

[0087] In this embodiment of the invention, a predictive potential accumulation mechanism is introduced into the LIF neurons of the pulse propagation layer. When performing the membrane potential update at the current moment, the neuron no longer uses a fixed constant coefficient to accumulate the weight of the input pulse, but dynamically adjusts the response coefficient of the current input pulse according to the changing state in the recording unit. At the same time, local plasticity regulation based on time difference is introduced into the synaptic connection to locally regulate the transmission intensity. When the application scenario experiences sudden changes in illumination or deformation of the target shape, the network can rapidly enhance effective feature channels and suppress noise channels locally.

[0088] In an exemplary embodiment of the present invention, the step of extracting the local spatiotemporal features of the event block to obtain the event tensor includes: Within each event block, the direction of event change is determined according to the time sequence of the event signals, and a spatiotemporal correspondence is established based on the time interval and spatial position difference between adjacent event signals. Based on the spatiotemporal correspondence, the event change trend information is statistically analyzed, and the event change trend information is encoded according to the time dimension and the spatial dimension to generate the local spatiotemporal features; The local spatiotemporal features are formatted to generate the event tensor.

[0089] In this embodiment of the invention, among two adjacent event signals, the temporal sequence and spatial direction of change of the later event signal relative to the earlier event signal are calculated. When the time stamp of the later event signal is later than that of the earlier event signal and the spatial coordinates are displaced, the direction of event change is recorded as the displacement direction from the earlier event signal to the later event signal. A spatiotemporal correspondence is established based on the time interval and spatial position difference between adjacent event signals. Specifically, within each event block, the time interval and spatial distance between event signals are determined according to the direction of event change. When both the time interval and spatial distance between event signals are within a preset range, it is determined that the event signal pair has a spatiotemporal correspondence, and the relationship is registered in the event block. If the time interval or spatial distance exceeds the corresponding range, it is determined to be different spatiotemporal segments, and they are recorded separately.

[0090] Based on the spatiotemporal correspondence, statistical information on event change trends is obtained, and this information is encoded according to the time and space dimensions to form a data structure representing local spatiotemporal characteristics. This data structure is then uniformly formatted to form an event tensor.

[0091] During the aggregation process, the event signals within each event block are arranged in the order of their occurrence, and their spatial and temporal adjacency relationships are recorded.

[0092] In an exemplary embodiment of the present invention, the step of performing spatiotemporal mapping on the pulse distribution to obtain a target event region, and generating a target detection result based on the feature representation of the target event region, includes: Based on the target detection results of multiple consecutive video signals, candidate matching pairs are established. Calculate the consistency index of each candidate matching pair in terms of spatial location and pulse frequency; Based on the consistency index, the matching relationship of the target objects in the video signal is determined; Based on the matching relationship, a trajectory sequence for each target object is generated.

[0093] In this embodiment of the invention, the detection result set within a continuous time slice (frame) is read, and candidate matching pairs are established according to the timestamp order contained in the target detection results. That is, the time interval between each target object in the previous time slice and all target objects in the next time slice is calculated, and target object pairs with time intervals within a preset range are selected as candidate matching pairs.

[0094] The consistency index of candidate matching pairs in terms of spatial location and pulse frequency is calculated. Candidate pairs are sorted from highest to lowest consistency index to determine the matching relationships of the same target object in adjacent time slices. Based on the determined matching relationships, the trajectory records of each target object are updated, and the new spatial coordinate information is appended to the corresponding trajectory sequence.

[0095] When a target object is detected as not matching successfully within a certain number of consecutive time slices, trajectory recording is terminated. When a new unmatched target object is detected, a new identification number is assigned to it and a new trajectory record is established.

[0096] To verify the feasibility of the technical solution provided by this invention in implementation, the technical solution was applied to a multi-target dynamic recognition experiment in a smart warehouse sorting scenario. The experimental object was a simulated sorting conveyor line, on which packages, parts and containers of various shapes and materials were randomly placed. The objects moved at a speed of 0.5–1.2 m / s, and the video acquisition device was a conventional industrial camera (resolution 1920×1080, frame rate 60fps).

[0097] During the video acquisition phase, event signals are generated from the real-time acquired color video signals. First, the video luminance component is extracted, and median filtering and local histogram equalization are performed to suppress noise and balance luminance distribution. Then, pixel-level luminance differential is performed on consecutive frames. When the luminance change exceeds a preset luminance change threshold, an event signal containing pixel coordinates, luminance change polarity, and time information is generated. The event signals are organized chronologically to form a sparse event stream sequence and queued in a buffer according to time priority to ensure real-time event processing.

[0098] During event aggregation, event signals are clustered into event blocks using time windows as units. The initial time window is set to 10 milliseconds and is adaptively adjusted according to the event triggering frequency. For example, if the event triggering frequency increases by more than 15% compared to the previous window, the window length is shortened to 0.8 times the original value; if the triggering frequency decreases by more than 15%, the window length is extended to 1.2 times the original value. Within each time window, event signals are aggregated according to temporal and spatial proximity, recording the spatiotemporal adjacency relationships of events within the event block, thus forming an aggregated structure that can represent local dynamic changes.

[0099] During the feature extraction stage, the time interval and spatial displacement of adjacent event signals within each event block are calculated to establish a spatiotemporal correspondence. When the time interval is less than 3 milliseconds and the spatial displacement is less than 3 pixels, the event signal pair is determined to belong to the same dynamic segment. The change trend information of all dynamic segments is statistically analyzed, and these change trends are encoded into three-dimensional event tensors according to the time and spatial dimensions to represent the local spatiotemporal change patterns and provide input for subsequent spiking neural network model inference.

[0100] In the inference phase of the spiking neural network model, the event tensor is input into the model, which consists of leaky integral firing neurons. Each neuron retains the direction of change from the previous time step when updating its membrane potential. When an increase in membrane potential is detected for two consecutive time steps, the response coefficient of the input pulse signal is increased, enabling the neuron to reach the firing condition more quickly when events increase. When a decreasing trend is detected, the response coefficient is decreased to avoid false triggering. This predictive potential accumulation mechanism allows neurons to fire pulses in advance when sudden events occur, shortening the overall detection response time.

[0101] In synaptic connections, a time-difference-based local plasticity adjustment mechanism is introduced. When the input pulse signal arrives 1 to 2 milliseconds earlier than the output pulse signal, the transmission strength of the synaptic connection is enhanced by approximately 3%; when the input pulse signal lags by the same amount of time, its strength is weakened by 3%; and when the time difference is close to zero, it remains unchanged. This local plasticity adjustment mechanism can dynamically adjust synaptic weights according to the pulse time difference, enabling the model to automatically converge in long-term operation, reducing temporal interference, and improving the consistency of responses between neurons.

[0102] When the number of event pulses per unit time exceeds the wake-up threshold, the detection network is automatically activated. The detection network performs connectivity analysis on the target event region, dividing the target event region composed of adjacent pixels into candidate target regions. The mean pulse intensity of each candidate target region within the current time window is calculated, and the confidence level is estimated based on this mean and its stability. A high confidence level is assigned when the mean pulse intensity is higher than 0.7 and the fluctuation amplitude is less than 10% within three consecutive frames; a low confidence level is assigned when the mean is lower than 0.5 or the fluctuation amplitude is greater than 20%; all other cases are assigned a medium confidence level. The final output includes the target category, confidence value, and spatial coordinates.

[0103] During target tracking, target detection results from consecutive time slices are read, and candidate matching pairs are established based on timestamps. If the spatial distance between the two target detection results of a candidate matching pair is less than 10 pixels and the pulse frequency difference is less than 5%, they are considered to correspond to the same target object. The trajectory record is updated according to the matching results. Recording is terminated for target objects that fail to match for three consecutive frames. Newly appearing target objects are automatically assigned an identification number and a new trajectory is established.

[0104] During power consumption control, the event trigger frequency is counted every 30 seconds. When the event trigger frequency exceeds 1200 events / second, the preset brightness change threshold is increased by 10%; when the event trigger frequency is below 800 events / second, the preset brightness change threshold is decreased by 10%. Through periodic adjustments, the system reduces energy consumption under high load scenarios and maintains detection sensitivity under low activity scenarios, thereby achieving adaptive power consumption control during the event generation phase.

[0105] Under the same hardware conditions, a comparative experiment was conducted using traditional frame detection, event detection, and the solution provided by this invention. The experimental results are shown in Table 1 below: Table 1

[0106] As shown in Table 1, the method of this invention reduces detection latency by about one-third, improves accuracy by about 4 percentage points, and reduces power consumption by about 12% under the same hardware conditions. The predictive potential accumulation mechanism enables neurons to respond in advance to sudden events, significantly improving the real-time performance of detection; the time difference-based local plasticity regulation mechanism effectively enhances the temporal consistency of pulse propagation, making the detection results more stable and the target trajectory more continuous. This method is applicable to multi-target video intelligent recognition scenarios such as intelligent sorting, industrial inspection, and robot vision, and has good practicality and promotional value.

[0107] The following describes the AI-based multi-target video intelligent detection device provided by the present invention. The AI-based multi-target video intelligent detection device described below corresponds to the AI-based multi-target video intelligent detection method described above. It should be noted that the device provided in the following embodiments belongs to the same concept as the method provided in the above embodiments, and the specific manner in which each module and unit performs its operation has been described in detail in the method embodiments, and will not be repeated here.

[0108] In one exemplary embodiment of the present invention, please refer to Figure 2 , Figure 2 An exemplary embodiment of an artificial intelligence-based multi-target video intelligent detection device is shown, comprising the following modules: Acquisition module 210 is configured to acquire video signals and organize the event signals in the video signals into a sparse event stream sequence according to the time sequence. Aggregation module 220 is configured to group and aggregate the event signals in the sparse event stream sequence based on a preset time window and spatial location association to form event blocks; Extraction module 230 is configured to extract the local spatiotemporal features of the event block to obtain an event tensor; Input module 240 is configured to input the event tensor into a spiking neural network model to obtain a pulse distribution; wherein, the spiking neural network model is provided with a predictive potential accumulation mechanism and a local plasticity regulation mechanism, the predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment, and the local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal; The detection module 250 is configured to perform spatiotemporal mapping on the pulse distribution to obtain a target event region, and generate a target detection result based on the feature representation of the target event region.

[0109] In an exemplary embodiment of the present invention, the input module 240 includes: A submodule is established and configured to create a change state recording unit for each neuron in the spiking neural network model, and to store the direction and magnitude of the increase or decrease of the membrane potential of the neuron at the previous moment through the change state recording unit; The first adjustment submodule is configured to adjust the response coefficient corresponding to the current input pulse signal of each neuron according to the direction of increase or decrease of the membrane potential when performing membrane potential update; wherein, if the membrane potential is determined to be increasing based on the direction of increase or decrease of the membrane potential, the response coefficient is increased; if the membrane potential is determined to be decreasing based on the direction of increase or decrease of the membrane potential, the response coefficient is decreased.

[0110] In an exemplary embodiment of the present invention, the input module 240 includes: The acquisition submodule is configured to acquire the arrival time difference between the input pulse signal and the output pulse signal in the synaptic connection; The second adjustment submodule is configured to dynamically adjust the transmission strength of the synaptic connection based on the arrival time difference, wherein when the arrival time of the input pulse signal is earlier than the arrival time of the output pulse signal, the transmission strength is increased, and when the arrival time of the input pulse signal is later than the arrival time of the output pulse signal, the transmission strength is decreased. The update submodule is configured to synchronously update the pulse distribution output by the spiking neural network model based on the adjusted transmission intensity.

[0111] In an exemplary embodiment of the present invention, the detection module 250 includes: The statistics submodule is configured to count the number of event pulses per unit time and compare the number of event pulses with a preset wake-up threshold. The wake-up submodule is configured to wake up the detection network in a dormant state if the number of event pulses is greater than or equal to the wake-up threshold. The first generation submodule is configured to generate the target detection result, which includes the target category, confidence level, and spatial coordinates, through the detection network based on the feature representation of the target event region.

[0112] In an exemplary embodiment of the present invention, the length of the preset time window is adjusted based on the event triggering frequency; wherein, when the event triggering frequency increases, the length of the preset time window is shortened, and when the event triggering frequency decreases, the length of the preset time window is extended.

[0113] In one exemplary embodiment of the present invention, the artificial intelligence-based multi-target video intelligent detection device further includes: The statistics module is configured to count the number of times the event signal is triggered according to a preset time period, and calculate the event trigger frequency based on the number of triggers; The enhancement module is configured to increase the signal threshold for generating event signals if the event triggering frequency is greater than or equal to a preset triggering rate threshold. The reduction module is configured to reduce the signal threshold for generating event signals if the event triggering frequency is less than the preset triggering rate threshold.

[0114] In an exemplary embodiment of the present invention, the detection module 250 includes: The first determining submodule is configured to perform time and space mapping on the pulse distribution to determine the active event region that is continuously activated within a continuous time slice as the target event region; The analysis submodule is configured to perform connectivity analysis on the target event region and divide the target event region into multiple candidate target regions based on the connectivity analysis results. The feature overlay submodule is configured to overlay the event signals within each candidate target region according to spatial and temporal order to generate a region-level feature representation; The second generation submodule is configured to generate the target detection result, which includes the target category, confidence level, and spatial coordinates, based on the region-level feature representation.

[0115] In an exemplary embodiment of the present invention, the spiking neural network model includes an input encoding layer, a pulse propagation layer, and a pulse decoding layer; the input module 240 includes: The conversion submodule is configured to convert the time-series information in the event tensor into a pulse sequence based on the input encoding layer; wherein the pulse sequence maintains the temporal order of the event signals. The execution submodule is configured to perform accumulation and firing operations on the temporal relationship of the input pulse signal through each neuron in the pulse propagation layer based on the pulse sequence, to obtain the output pulse signal; The aggregation submodule is configured to perform time-series aggregation on the output pulse signal based on the pulse decoding layer to generate the pulse distribution.

[0116] In an exemplary embodiment of the present invention, the acquisition module 210 includes: The processing submodule is configured to perform noise reduction and brightness equalization processing on the video signal; The third generation submodule is configured to generate an event signal containing pixel coordinates, brightness change polarity, and time information if the brightness change amplitude of the same pixel is detected to be greater than or equal to a preset brightness change threshold in adjacent video frames of the processed video signal. A submodule is configured to arrange all generated event signals in chronological order to construct the sparse event stream sequence.

[0117] In an exemplary embodiment of the present invention, the extraction module 230 includes: The second determining submodule is configured to determine the direction of event change in each event block according to the time sequence of the event signals, and establish a spatiotemporal correspondence based on the time interval and spatial position difference between adjacent event signals. The encoding submodule is configured to statistically analyze event change trend information based on the spatiotemporal correspondence, encode the event change trend information according to the time dimension and the spatial dimension, and generate the local spatiotemporal features. The formatting processing submodule is configured to perform formatting processing on the local spatiotemporal features to generate the event tensor.

[0118] In one exemplary embodiment of the present invention, the artificial intelligence-based multi-target video intelligent detection device further includes: Establish a module and configure it to create candidate matching pairs based on the target detection results of multiple consecutive video signals; The calculation module is configured to calculate the consistency index of each candidate matching pair in terms of spatial location and pulse frequency; The determination module is configured to determine the matching relationship of target objects in the video signal based on the consistency index; The generation module is configured to generate trajectory sequences for each of the target objects based on the matching relationship.

[0119] Figure 3 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 3As shown, the electronic device may include a processor 310, a communications interface 320, a memory 330, and a communication bus 340, wherein the processor 310, the communications interface 320, and the memory 330 communicate with each other via the communication bus 340. The processor 310 can call logical instructions in the memory 330 to execute an artificial intelligence-based multi-target video intelligent detection method, which includes: acquiring video signals and organizing the event signals in the video signals into a sparse event stream sequence according to time sequence; Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks; Extract the local spatiotemporal features of the event block to obtain the event tensor; The event tensor is input into a spiking neural network model to obtain a pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity regulation mechanism. The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment. The local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal. The target event region is obtained by performing spatiotemporal mapping on the pulse distribution, and the target detection result is generated based on the feature representation of the target event region.

[0120] Furthermore, the logical instructions in the aforementioned memory 330 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, essentially, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0121] On the other hand, the present invention also provides a computer program product, the computer program product including a computer program, the computer program being able to be stored on a non-transitory computer-readable storage medium, and when the computer program is executed by a processor, the computer is able to execute the artificial intelligence-based multi-target video intelligent detection method provided by the above methods, the method including: acquiring video signals, and organizing the event signals in the video signals into a sparse event stream sequence according to the time sequence; Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks; Extract the local spatiotemporal features of the event block to obtain the event tensor; The event tensor is input into a spiking neural network model to obtain a pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity regulation mechanism. The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment. The local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal. The target event region is obtained by performing spatiotemporal mapping on the pulse distribution, and the target detection result is generated based on the feature representation of the target event region.

[0122] In another aspect, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the artificial intelligence-based multi-target video intelligent detection method provided by the above methods, the method comprising: acquiring video signals and organizing event signals in the video signals into a sparse event stream sequence according to time sequence; Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks; Extract the local spatiotemporal features of the event block to obtain the event tensor; The event tensor is input into a spiking neural network model to obtain a pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity regulation mechanism. The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment. The local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal. The target event region is obtained by performing spatiotemporal mapping on the pulse distribution, and the target detection result is generated based on the feature representation of the target event region.

[0123] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0124] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0125] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target video intelligent detection method based on artificial intelligence, characterized in that, include: Acquire video signals and organize the event signals in the video signals into a sparse event stream sequence according to time order; Based on a preset time window and spatial location correlation, the event signals in the sparse event stream sequence are grouped and aggregated to form event blocks; Extract the local spatiotemporal features of the event block to obtain the event tensor; The event tensor is input into a spiking neural network model to obtain a pulse distribution. The spiking neural network model is equipped with a predictive potential accumulation mechanism and a local plasticity regulation mechanism. The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment. The local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal. The target event region is obtained by performing spatiotemporal mapping on the pulse distribution, and the target detection result is generated based on the feature representation of the target event region; The predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential changes at the target historical moment of the neuron, including: A change state recording unit is established for each neuron in the spiking neural network model, and the change state recording unit stores the direction and magnitude of the increase or decrease of the membrane potential of the neuron at the previous moment; When performing membrane potential updates, the response coefficients corresponding to the current input pulse signals of each neuron are adjusted according to the direction of increase or decrease of the membrane potential; wherein, if the membrane potential is determined to be increasing based on the direction of increase or decrease of the membrane potential, the response coefficient is increased, and if the membrane potential is determined to be decreasing based on the direction of increase or decrease of the membrane potential, the response coefficient is decreased; wherein, the response coefficient includes the cumulative step size of the input pulse signal.

2. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The local plasticity adjustment mechanism adjusts the transmission strength of synaptic connections based on the arrival time difference between the input pulse signal and the output pulse signal, including: Obtain the arrival time difference between the input pulse signal and the output pulse signal in the synaptic connection; The transmission strength of the synaptic connection is dynamically adjusted based on the arrival time difference, wherein the transmission strength is increased when the arrival time of the input pulse signal is earlier than the arrival time of the output pulse signal, and the transmission strength is decreased when the arrival time of the input pulse signal is later than the arrival time of the output pulse signal. Based on the adjusted transmission intensity, the pulse distribution output by the spiking neural network model is updated synchronously.

3. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The generation of target detection results based on the feature representation of the target event region includes: The number of event pulses per unit time is counted, and the number of event pulses is compared with a preset wake-up threshold. If the number of event pulses is greater than or equal to the wake-up threshold, the detection network in a dormant state is woken up; Based on the feature representation of the target event region, the detection network generates the target detection result, which includes the target category, confidence level, and spatial coordinates.

4. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The length of the preset time window is adjusted based on the event triggering frequency; wherein, when the event triggering frequency increases, the length of the preset time window is shortened, and when the event triggering frequency decreases, the length of the preset time window is extended.

5. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The method further includes: The number of times the event signal is triggered is counted according to a preset time period, and the event triggering frequency is calculated based on the number of triggers. If the event trigger frequency is greater than or equal to a preset trigger rate threshold, the signal threshold for generating the event signal is increased. If the event triggering frequency is less than the preset triggering rate threshold, the signal threshold for generating the event signal is reduced.

6. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The process of obtaining a target event region by performing spatiotemporal mapping on the pulse distribution, and generating a target detection result based on the feature representation of the target event region, includes: The pulse distribution is mapped in time and space to determine the active event region that is continuously activated within a continuous time slice as the target event region; A connectivity analysis is performed on the target event region, and based on the connectivity analysis results, the target event region is divided into multiple candidate target regions; According to spatial and temporal order, the event signals in each candidate target region are superimposed to generate a region-level feature representation; Based on the region-level feature representation, the target detection result containing target category, confidence level, and spatial coordinates is generated.

7. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The spiking neural network model includes an input encoding layer, a pulse propagation layer, and a pulse decoding layer; The step of inputting the event tensor into a spiking neural network model to obtain the pulse distribution includes: Based on the input encoding layer, the time-series information in the event tensor is transformed into a pulse sequence; wherein the pulse sequence preserves the temporal order of the event signals; Based on the pulse sequence, the output pulse signal is obtained by performing accumulation and firing operations on the temporal relationship of the input pulse signal through each neuron in the pulse propagation layer. Based on the pulse decoding layer, the output pulse signal is time-series aggregated to generate the pulse distribution.

8. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The step of organizing the event signals in the video signal into a sparse event stream sequence according to time order includes: The video signal is subjected to noise reduction and brightness equalization processing; In adjacent video frames of the processed video signal, if the brightness change of the same pixel is detected to be greater than or equal to a preset brightness change threshold, an event signal containing pixel coordinates, brightness change polarity, and time information is generated. All generated event signals are arranged in chronological order to construct the sparse event stream sequence.

9. The multi-target video intelligent detection method based on artificial intelligence according to claim 1, characterized in that, The step of extracting the local spatiotemporal features of the event block to obtain the event tensor includes: Within each event block, the direction of event change is determined according to the time sequence of the event signals, and a spatiotemporal correspondence is established based on the time interval and spatial position difference between adjacent event signals. Based on the spatiotemporal correspondence, the event change trend information is statistically analyzed, and the event change trend information is encoded according to the time dimension and the spatial dimension to generate the local spatiotemporal features; The local spatiotemporal features are formatted to generate the event tensor.

10. The multi-target video intelligent detection method based on artificial intelligence according to any one of claims 1 to 9, characterized in that, After obtaining the target event region by performing spatiotemporal mapping on the pulse distribution, and generating the target detection result based on the feature representation of the target event region, the method further includes: Based on the target detection results of multiple consecutive video signals, candidate matching pairs are established. Calculate the consistency index of each candidate matching pair in terms of spatial location and pulse frequency; Based on the consistency index, the matching relationship of the target objects in the video signal is determined; Based on the matching relationship, a trajectory sequence for each target object is generated.

11. A multi-target video intelligent detection device based on artificial intelligence, characterized in that, include: The acquisition module is configured to acquire video signals and organize the event signals in the video signals into a sparse event stream sequence according to the time sequence. The aggregation module is configured to group and aggregate the event signals in the sparse event stream sequence based on a preset time window and spatial location association to form event blocks. The extraction module is configured to extract the local spatiotemporal features of the event block to obtain the event tensor; The input module is configured to input the event tensor into a spiking neural network model to obtain a pulse distribution; wherein, the spiking neural network model is provided with a predictive potential accumulation mechanism and a local plasticity regulation mechanism, the predictive potential accumulation mechanism adjusts the response coefficient of the input pulse signal based on the membrane potential change state of the neuron at the target historical moment, and the local plasticity regulation mechanism adjusts the transmission strength of the synaptic connection based on the arrival time difference between the input pulse signal and the output pulse signal; The detection module is configured to perform spatiotemporal mapping on the pulse distribution to obtain a target event region, and generate a target detection result based on the feature representation of the target event region; The input module includes: A submodule is established and configured to create a change state recording unit for each neuron in the spiking neural network model, and to store the direction and magnitude of the increase or decrease of the membrane potential of the neuron at the previous moment through the change state recording unit; The first adjustment submodule is configured to adjust the response coefficient corresponding to the current input pulse signal of each neuron according to the direction of increase or decrease of the membrane potential when performing membrane potential update; wherein, if the membrane potential is determined to be increasing based on the direction of increase or decrease of the membrane potential, the response coefficient is increased; if the membrane potential is determined to be decreasing based on the direction of increase or decrease of the membrane potential, the response coefficient is decreased; wherein, the response coefficient includes the cumulative step size of the input pulse signal.

12. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the AI-based multi-target video intelligent detection method as described in any one of claims 1 to 10.

13. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the AI-based multi-target video intelligent detection method as described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the AI-based multi-target video intelligent detection method as described in any one of claims 1 to 10.

Images