Visual high-speed camera system based on event triggering

Abstract

The invention discloses a visual high-speed camera system based on event triggering. The visual high-speed camera system comprises a control module, an image module, a triggering algorithm module and a storage module which are connected with one another, according to the method and the device, a plurality of triggering conditions such as motion edge, brightness mutation and target detection are combined, and a low-delay event triggering mechanism is adopted, so that the system can quickly respond at a critical moment and start high-speed recording. The optimized system can efficiently capture instant changes in a dynamic scene, and the success rate of image capture and the response speed of the system are remarkably improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of image acquisition and processing, and in particular relates to a vision fusion high-speed camera system based on event triggering. Background Art

[0002] Traditional high-speed cameras typically use a fixed frame rate continuous shooting mode, resulting in a large amount of redundant data and placing significant strain on storage and bandwidth. Furthermore, existing technologies lack the ability to detect critical events in real time, making it difficult to respond quickly in complex dynamic scenes. For example, industrial inspections require capturing transient defects, but traditional systems, due to high latency and low storage efficiency, can miss key frames.

[0003] In existing technologies, some solutions attempt to optimize data collection through sensor fusion or trigger mechanisms, but the following problems still exist: First, event detection latency is high: software-based analysis algorithms cannot meet millisecond-level response requirements; second, storage bandwidth is limited: high-speed data flow causes storage devices to be easily overloaded, causing frame drops or data loss; finally, the trigger condition is single: relying only on motion detection or brightness changes, it cannot adapt to the needs of multiple scenarios. Therefore, how to solve these problems is the problem we are currently facing. Summary of the Invention

[0004] In response to the shortcomings of the above technologies, the present invention provides a visual fusion high-speed camera system based on event triggering, which realizes rapid capture of events by setting a trigger algorithm module, thereby significantly improving image capture efficiency.

[0005] To achieve the above object, the present invention provides a visual high-speed camera system based on event triggering, comprising a control module, an image module, a trigger algorithm module and a storage module connected to each other;

[0006] The control module presets the trigger conditions and sets the acquisition time of the image module and the storage parameters of the storage module;

[0007] The image module collects high-definition images, obtains event information in the scene, and converts the event information into a data stream and transmits it to the trigger algorithm module;

[0008] The trigger algorithm module pre-processes the data stream, then detects the pre-processed data stream, identifies the target event based on image feature extraction and pattern recognition algorithms, and transmits the recognition signal to the image module and the storage module;

[0009] After receiving the recognition signal, the storage module stores the information collected by the image module.

[0010] Preferably, the trigger algorithm module has an event-driven explosion detection algorithm built in, comprising the following steps:

[0011] S1: Asynchronous event streams are collected by event vision sensors (EVS), where each event is represented by (x i ,y i ,t i ,p i ), x i and y i is the pixel coordinate, t i The event trigger timestamp, p i Indicates event polarity;

[0012] S2: Preprocessing the event stream, including time threshold filtering, spatial neighborhood check and polarity consistency check, to filter out isolated events and electronic noise;

[0013] S3: Constructing a first event accumulation map (FEAM) and a second event accumulation map (SEAM) based on the sliding time window, wherein the FEAM covers the time period [t-ΔT, t] and the SEAM covers the time period [t, t+ΔT], where ΔT is a fixed time length;

[0014] S4: Calculate the difference map D(x,y) between FEAM and SEAM, and extract the area with sudden increase in event density as the candidate target;

[0015] S5: performing event clustering and optical flow estimation on the candidate target, calculating the radially diffused optical flow vector field based on the clustering result, and analyzing the consistency of the motion direction and speed;

[0016] S6: Difference map peak D max , optical flow energy V avg and radial consistency cos(θ avg ) is weighted fusion to obtain the comprehensive score S = α·D max +β·V avg +γ·cos(θ avg ), where α, β, and γ are weight coefficients;

[0017] S7: Determine whether there is an explosion based on the preset dynamic threshold. If the conditions are met:

[0018] D max >K D μ D , where μ D is the historical average difference value;

[0019] V avg >V threshold ;

[0020] cos(θ avg )>C threshold ;

[0021] Comprehensive score S>S threshold ;

[0022] It is determined that an explosion occurred in the target area.

[0023] Preferably, the method for generating the event accumulation graph comprises the following steps:

[0024] S1: Initialize a grid of size 768 x 604, corresponding to the sensor resolution;

[0025] S2: The events in the time window are weighted and accumulated to the corresponding pixel position according to polarity, and the weight w(e i ) can be a fixed weight or a time-decayed weight w(e i )=exp(-λ(tt i )), where λ = 0.1us -1 .

[0026] Preferably, the calculation of the optical flow vector field includes:

[0027] a: Spatial gradient calculation:

[0028] - Apply the Scharr operator to the event accumulation map to approximate the spatial gradient and obtain the horizontal gradient I x and vertical gradient I y :

[0029] I x =Scharr(D,dx=1,dy=0),I y =Scharr(D,dx=0,dy=1)

[0030] The Scharr kernel coefficient is:

[0031] G x =[-3,0,3;-10,0,10;-3,0,3],G y =[-3,-10,-3;

[0032] 0,0,0;3,10,3]

[0033] b: Time gradient approximation:

[0034] - Calculate the temporal gradient using the timestamp and polarity of the event:

[0035]

[0036] c: Lucas-Kanade optical flow solution:

[0037] -Construct the equation system in the neighborhood window W of the feature point (x0, y0):

[0038]

[0039] in:

[0040] -A ij =I x (x j ,y j ) 2 +I y (x j ,y j ) 2 ;

[0041] -b j =I x (x j ,y j )I t (x j ,y j )+I y (x j ,y j )I t (x j ,y j );

[0042] -Final solution:

[0043]

[0044] d: Calculate the centroid displacement of the motion area after event clustering

[0045] e: Fitting the local motion direction using the least squares method, assuming the explosion spreads radially;

[0046] f: Based on clustering energy E i (Number of events × polarity distribution consistency) is energy-weighted averaged to obtain V avg .

[0047] As an advantage, it also includes a post-processing step:

[0048] Time-space consistency check: The explosion characteristics must be met in M = 3 consecutive time windows, and the explosion area must be larger than A. min =100px 2

[0049] Preferably, the pretreatment step comprises:

[0050] S1: Time threshold filtering: filtering duration is less than T min = Isolated event clusters of 1ms;

[0051] S2: Spatial neighborhood check: only the areas within the surrounding 4x 4 pixels containing at least 5 events are retained;

[0052] S3: Polarity consistency check: If the frequency of alternating positive and negative polarity in the K x K window exceeds the threshold F p = 0.5, marked as noise and removed.

[0053] Preferably, when the trigger algorithm module obtains a trigger signal from the inspection unit, the image module collects image information, and at the same time the storage module stores the information collected by the image module, switching from the preview state to the cache state, and all the information is stored in the cache area. When the cache area is full, the asynchronous transmission method is used to transmit the information in batches to the long-term storage device for retention.

[0054] Preferably, an adjustment unit is also provided in the control module, which dynamically adjusts the response priority according to the urgency and importance; at the same time, based on the changes in the real-time data stream, it automatically adjusts the detection threshold, cache size, transmission bandwidth and other parameters to ensure stable operation under various workloads.

[0055] The beneficial effects of the present invention are as follows: the event-triggered visual high-speed camera system provided by the present application integrates motion edge detection, brightness mutation detection and specific area target detection through a multi-condition trigger mechanism equipped with an event trigger algorithm, and realizes rapid capture of events through a special algorithm; in addition, dynamic changes in the scene are detected at a high frame rate, event stream data is output, and frame loss is avoided. Finally, optimization strategy adjustment can be performed, and the cache size and transmission bandwidth are dynamically adjusted according to hardware resources, while optimizing the processing order based on the urgency of the event. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] Attachment Figure 1 This is a flowchart of the steps of this application. DETAILED DESCRIPTION

[0057] In order to more clearly describe the present invention, the present invention is further described below in conjunction with the accompanying drawings. Of course, the protection scope of the present invention is not limited to this. Simple replacements that can be made by those skilled in the art without paying creative work fall within the protection scope of the present invention.

[0058] See also Figure 1 The present invention discloses a visual high-speed camera system based on event triggering, comprising a control module, an image module, a trigger algorithm module and a storage module connected to each other;

[0059] The control module presets the trigger conditions and sets the acquisition duration of the image module and the storage parameters of the storage module. The image module captures high-definition images, obtains event information in the scene, and converts the event information into a data stream and transmits it to the trigger algorithm module. The trigger algorithm module preprocesses the data stream and then detects the preprocessed data stream. Based on image feature extraction and pattern recognition algorithms, it identifies the target event and transmits the recognition signal to the image module and the storage module. After receiving the recognition signal, the storage module stores the information collected by the image module. In the specific implementation process, an active pixel sensor (APS) and an event-driven sensor (EVS) are combined. The APS sensor is responsible for acquiring high-definition images, and the EVS sensor is responsible for high-speed perception of image changes, monitoring events in the scene, and providing real-time feedback based on the set trigger conditions. The relevant recognition algorithm is set in the EVS sensor to determine whether the recording condition is triggered. At the same time, a high-speed cache storage unit is provided in the storage module, and asynchronous transmission is used to transfer the data in the cache storage unit (SSD) to a hard disk or other location for permanent storage.

[0060] The trigger algorithm module includes a processing unit and a detection unit. The processing unit preprocesses the data stream, including time threshold filtering, spatial neighborhood check and polarity consistency check, filtering isolated events and electronic noise. The data stream obtained after preprocessing is transmitted to the detection unit.

[0061] The input event stream format is (x i ,y i ,t i ,p i ), x i and y i is the pixel coordinate, t i The event trigger timestamp, p i Indicates event polarity. Preprocessing includes: temporal threshold filtering: filtering out isolated event clusters with a duration of less than 1ms; spatial neighborhood check: retaining regions with at least 5 events within a 4x4 pixel area; and polarity consistency check: removing noise with a polarity alternation frequency greater than 0.5.

[0062] In explosion detection, a combination of difference analysis between the first event accumulation map (FEAM) and the second event accumulation map (SEAM) and event-driven optical flow estimation enables efficient detection of explosion events. Specifically, the FEAM and SEAM are constructed using a sliding time window to extract regions with sudden increases in event density as candidate targets. Furthermore, the radially diffused optical flow vector field is calculated based on event clustering to analyze the consistency of motion direction and speed. Finally, a weighted fusion of multi-dimensional features, such as the difference map peak, optical flow energy, and radial consistency, is combined with a dynamic threshold to determine whether an explosion has occurred.

[0063] Event accumulation graph generation: FEAM is generated in the first period [t-ΔT, t], and SEAM is generated in the second period [t, t+ΔT], where ΔT = 10ms. The accumulation graph calculation formula is:

[0064]

[0065] The weight w(e i ) can be a fixed weight or a time-decayed weight w(e i )=exp(-λ(tt i )).

[0066] Difference map and candidate target extraction: Difference map D(x,y) = |SEAM-FEAM|, the explosion area is represented by D(x,y)>K D μ D (K D =3).

[0067] The event-driven optical flow estimation algorithm steps include Scharr spatial gradient calculation, temporal gradient approximation, and Lucas-Kanade optical flow solution.

[0068] Use the Scharr operator instead of the traditional Sobel operator to improve the accuracy of spatial gradient calculation: horizontal gradient I x Using core G x =[-3,0,3;-10,0,10;-3,0,3]; vertical gradient I y Using core G y =[-3,-10,-3; 0,0,0; 3,10,3].

[0069] Binding event polarity p i and time interval Δt, defining the time gradient I t (x,y):

[0070]

[0071] This gradient characterizes the instantaneous rate of change of brightness.

[0072] In the feature point neighborhood window W (default 5x 5 pixels), the optical flow vector (u, v) is solved by the least squares method: construct the matrix A and vector b, based on the spatial gradient I x ,I y and time gradient I t Solve the linear equation system A[u;v]=-b to obtain the direction and speed of motion. For the motion area after event clustering, calculate the center of mass displacement v based on optical flow estimation i , fitting the radial diffusion direction; energy-weighted optical flow formula:

[0073]

[0074] Among them E i is the clustering energy (number of events × polarity consistency).

[0075] The detection unit integrates the difference map peak, optical flow velocity, and radial consistency calculation results into features, makes a comprehensive judgment, and uses the comprehensive scoring formula as follows:

[0076] S=α·D max +β·V avg +γ·cos(θ avg )

[0077] Judgment condition: Difference map peak D max >K D μ D ; Optical flow velocity|V avg |>1000px / s; radial consistency cos(θ avg )>0.8; comprehensive score S>S threshold .

[0078] In addition, the detection unit will also post-process the detection results, including spatiotemporal consistency verification: 3 consecutive time windows meet the conditions and the coverage area is greater than 100px 2 ;

[0079] When the trigger algorithm module receives a trigger signal from the inspection unit, the imaging module begins capturing image information, while the storage module simultaneously stores the captured information. The system switches from preview mode to cache mode, storing all information in the cache. When the cache is full, it switches to asynchronous transmission, transferring the information in batches to a long-term storage device for storage. The control module also includes an adjustment unit that dynamically adjusts response priorities based on urgency and importance. It also automatically adjusts parameters such as detection thresholds, cache size, and transmission bandwidth based on changes in the real-time data stream to ensure stable operation under various workloads. In this embodiment, when the trigger condition is met, the APS sensor is rapidly activated for high-speed recording, achieving low latency and fast response, ensuring no key frames are lost. More specifically, under normal circumstances, both the APS and EVS sensors are in preview mode and do not save data. When an event is triggered, the APS and EVS sensors for the current frame are immediately switched to high-speed storage mode. This minimizes data stream switching latency, ensuring real-time acquisition. For high-speed image data, the system first stores the data in a cache (such as DDR memory). When the cache is full, the system uses asynchronous transmission to transfer data in batches to long-term storage devices such as hard drives and network storage to ensure data is not lost. Furthermore, data transmission occurs in parallel with image acquisition, and efficient asynchronous storage avoids data write delays, thereby improving system throughput. Finally, to ensure the system's rapid response and efficient operation in various scenarios, the control module can be optimized to dynamically adjust response priorities based on the urgency and importance of events. For example, if a sudden change in illumination over a large area is detected, the event processing priority can be increased.

[0080] The above disclosures are only several specific embodiments of the present invention, but the present invention is not limited thereto. Any changes that can be conceived by those skilled in the art should fall within the scope of protection of the present invention.

Claims

1. A visual high-speed camera system based on event triggering, characterized in that: It includes a control module, an image module, a trigger algorithm module and a storage module that are interconnected; The control module presets the trigger conditions and sets the acquisition time of the image module and the storage parameters of the storage module; The image module collects high-definition images, obtains event information in the scene, and converts the event information into a data stream and transmits it to the trigger algorithm module; The trigger algorithm module pre-processes the data stream, then detects the pre-processed data stream, identifies the target event based on image feature extraction and pattern recognition algorithms, and transmits the recognition signal to the image module and the storage module; After receiving the recognition signal, the storage module stores the information collected by the image module.

2. The event-triggered visual high-speed camera system according to claim 1, characterized in that: The trigger algorithm module has a built-in event-driven explosion detection algorithm, which includes the following steps: S1: Asynchronous event streams are collected by event vision sensors (EVS), where each event is represented by (x i ,y i ,t i ,p i ), x i and y i is the pixel coordinate, t i The event trigger timestamp, p i Indicates event polarity; S2: Preprocessing the event stream, including time threshold filtering, spatial neighborhood check and polarity consistency check, to filter out isolated events and electronic noise; S3: Constructing a first event accumulation map (FEAM) and a second event accumulation map (SEAM) based on the sliding time window, wherein the FEAM covers the time period [t-ΔT, t] and the SEAM covers the time period [t, t+ΔT], where ΔT is a fixed time length; S4: Calculate the difference map D(x,y) between FEAM and SEAM, and extract the area with sudden increase in event density as the candidate target; S5: performing event clustering and optical flow estimation on the candidate target, calculating the radially diffused optical flow vector field based on the clustering result, and analyzing the consistency of the motion direction and speed; S6: Difference map peak D max , optical flow energy V avg and radial consistency cos(θ avg ) is weighted fusion to obtain the comprehensive score S = α·D max +β·D avg +γ·cos(θ avg ), where α, β, and γ are weight coefficients; S7: Determine whether there is an explosion based on the preset dynamic threshold. If the conditions are met: D max >K D μ D , where μ D is the historical average difference value; V avg >V threshold ; cos(θ avg )>C threshold ; Comprehensive score S>S threshold ; It is determined that an explosion occurred in the target area.

3. The event-triggered high-speed camera system according to claim 2, characterized in that ,The method for generating the event accumulation graph includes the following steps: S1: Initialize a grid of size 768 x 604, corresponding to the sensor resolution; S2: The events in the time window are weighted and accumulated to the corresponding pixel position according to polarity, and the weight w(e i ) can be a fixed weight or a time-decayed weight w(e i )=exp(-λ(tt i )), where λ = 0.1us -1 .

4. The event-triggered visual high-speed camera system according to claim 2, characterized in that ,The calculation of the optical flow vector field includes: a: Spatial gradient calculation: - Apply the Scharr operator to the event accumulation map to approximate the spatial gradient and obtain the horizontal gradient I x and vertical gradient I y : I x =Scharr(D,dx=1,dy=0),I y =Scharr(D,dx=0,dy=1) The Scharr kernel coefficient is: G x =[-3,0,3;-10,0,10;-3,0,3],G y =[-3,-10,-3;0,0,0;3,10,3] b: Time gradient approximation: - Calculate the temporal gradient using the timestamp and polarity of the event: c: Lucas-Kanade optical flow solution: -Construct the equation system in the neighborhood window W of the feature point (x0, y0): in: -A ij =I x (x j ,y j ) 2 +I y (x j ,y j ) 2 ; -b j =I x (x j ,y j )I t (x j ,y j )+I y (x j ,y j )I t (x j ,y j ); -Final solution: d: Calculate the centroid displacement of the motion area after event clustering e: Fitting the local motion direction using the least squares method, assuming the explosion spreads radially; f: Based on clustering energy E i (Number of events × polarity distribution consistency) is energy-weighted averaged to obtain V avg .

5. The event-triggered visual high-speed camera system according to claim 2, characterized in that: Post-processing steps are also included: Time-space consistency check: The explosion characteristics must be met in M = 3 consecutive time windows, and the explosion area must be larger than A. min =100px 2。 6. The event-triggered visual high-speed camera system according to claim 1, characterized in that: The pre-processing step comprises: S1: Time threshold filtering: filtering duration is less than T min = Isolated event clusters of 1ms; S2: Spatial neighborhood check: only the areas within the surrounding 4x 4 pixels containing at least 5 events are retained; S3: Polarity consistency check: If the frequency of alternating positive and negative polarity in the K x K window exceeds the threshold F p = 0.5, marked as noise and removed.

7. The event-triggered visual high-speed camera system according to claim 1, characterized in that: When the trigger algorithm module receives the trigger signal from the inspection unit, the image module collects image information, and the storage module stores the information collected by the image module at the same time, switching from the preview state to the cache state. All information is stored in the cache area. When the cache area is full, the asynchronous transmission method is used to transmit the information in batches to the long-term storage device for retention.

8. The event-triggered visual high-speed camera system according to claim 1, characterized in that: The control module also has an adjustment unit that dynamically adjusts the response priority based on urgency and importance. At the same time, based on changes in real-time data streams, it automatically adjusts parameters such as detection thresholds, cache size, and transmission bandwidth to ensure stable operation under various workloads.

Images