An event camera optical flow and imu self-motion compensation based foreign object intrusion detection method and system

Abstract

The application discloses a foreign object intrusion detection method and system based on an event camera optical flow and IMU self-vehicle motion compensation, and relates to the technical field of vehicle-mounted active safety perception. The method comprises the following steps: multi-source data synchronous acquisition, wherein the multi-source data comprises an asynchronous event flow in front of a road and IMU data; time window division, space-time denoising and sparse optical flow estimation are performed on the event flow to obtain event-level observation optical flow; a self-vehicle angular velocity in an event camera coordinate system is estimated according to an IMU angular velocity and camera and IMU external parameters, a self-vehicle motion rotation induced optical flow is obtained, and residual optical flow is calculated; based on the residual optical flow, transverse dominant candidate events are screened, a ghost probe risk score is constructed by combining the number of candidate events, spatial aggregation degree, direction consistency, time burst degree and dangerous area weight, and a graded early warning result is output according to the risk score. The application effectively suppresses false positives caused by background transverse optical flow when a vehicle turns, changes lanes or has posture disturbance, and improves real-time performance and robustness.

Description

Technical Field

[0001] This invention relates to the fields of intelligent driving environment perception, event vision, inertial measurement fusion and vehicle active safety warning technology, and in particular to a foreign object intrusion detection method and system based on event camera optical flow and IMU vehicle motion compensation. Background Technology

[0002] Ghosting incidents are typically characterized by strong occlusion, rapid appearance, and short reaction time. Pedestrians, electric vehicles, or other road users often suddenly enter the vehicle's risk zone from parked vehicles, roadside facilities, wall edges, or areas obscured by greenery, easily leading to collisions. Existing methods mostly employ RGB cameras, millimeter-wave radar, lidar, or traditional frame-based visual algorithms for detection. However, frame-based cameras are prone to motion blur under high-speed movement and low-light conditions, while radar and lidar also suffer from insufficient resolution and response speed in identifying close-range, short-duration, and small targets.

[0003] Event cameras, with their microsecond-level temporal resolution and high dynamic range, can capture the instantaneous motion of lurking targets in vehicular scenes, making them suitable for detecting sudden lateral movements. However, lateral detection methods based on event camera optical flow typically assume the vehicle is traveling straight or in a stable state. When the vehicle turns, changes lanes, or experiences body vibration, its rotation introduces significant global lateral optical flow into the image plane, causing static background areas to also exhibit lateral motion. This leads to background events in the original observed optical flow being misidentified as lurking targets, significantly increasing the false alarm rate.

[0004] Therefore, there is an urgent need for a ghost detection scheme that can use IMU to estimate the vehicle's motion and subtract the vehicle's motion component from the event observation optical flow, so that the system can maintain stable, fast and low false alarm detection capabilities in scenarios such as straight driving, turning, lane changing and attitude disturbances. Summary of the Invention

[0005] The purpose of this invention is to provide a foreign object intrusion detection method and system based on event camera optical flow and IMU vehicle motion compensation, so as to solve the problems of existing ghost probe detection methods based on event camera optical flow being easily interfered with by background lateral optical flow when the vehicle is turning, changing lanes or the vehicle body posture is disturbed, thus generating false alarms.

[0006] To achieve the above objectives, the present invention adopts the following technical solution:

[0007] A foreign object intrusion detection method based on event camera optical flow and IMU vehicle motion compensation includes the following steps:

[0008] S1. Multi-source data synchronous acquisition and alignment: An event camera is used to acquire asynchronous event streams ahead of the road, and an IMU is used to acquire IMU data; the acquired event streams and IMU data are time-synchronized, and the synchronized data is divided into time windows. The event streams and IMU data are aligned according to the time windows based on the timestamps; the IMU data includes vehicle angular velocity and acceleration data;

[0009] S2. Event Sparse Optical Flow and IMU Angular Velocity Estimation: Spatiotemporal denoising is performed on the aligned event flow to obtain valid events. The pixel motion velocity corresponding to each event is calculated based on the valid events to obtain a sparse observation optical flow set. Based on the aligned IMU angular velocities in each time window and the extrinsic parameter relationship between the event camera and the IMU, the angular velocity of the vehicle in the event camera coordinate system in the current time window is estimated.

[0010] S3. Prediction of vehicle rotation-induced optical flow and calculation of residual optical flow: Based on the camera intrinsic parameters, event point coordinates and vehicle angular velocity, predict the image plane optical flow generated by the vehicle motion, denoted as vehicle rotation-induced optical flow; subtract the vehicle rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow;

[0011] S4. Lateral sudden motion detection based on residual optical flow: Perform residual lateral velocity screening and lateral dominance judgment on optical flow points in residual optical flow, extract lateral sudden motion candidate events, and obtain the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters based on the sudden motion candidate events.

[0012] S5. Ghost Probe Risk Scoring and Candidate Area Determination: Calculate the degree of time surge and the weight of dangerous areas. Based on this, combine the number of candidate events, residual lateral strength, spatial clustering and directional consistency parameters to calculate the comprehensive risk score. Determine the ghost probe candidate area based on the comprehensive risk score.

[0013] S6. Tiered Early Warning and Result Output: Based on the risk score, the risk level is determined according to the preset judgment rules, and corresponding safety measures are implemented based on the risk level determination results, including audible and visual alarms, vehicle prompts, platform recording, or active safety interfaces.

[0014] Furthermore, in step S1, hardware timestamps or software interpolation are used to synchronize the event stream with the IMU data, and the synchronized event stream is divided into a set of time windows according to a preset duration. The length of the time window is a fixed value or is adaptively adjusted according to the vehicle speed.

[0015] Furthermore, in step S2, the spatiotemporal denoising process for the aligned event stream includes: removing isolated events, flickering noise events, and abnormal events that do not satisfy neighborhood consistency.

[0016] Furthermore, the extrinsic parameter relationships in step S2 are obtained through pre-calibration, and the extrinsic parameter relationships are represented by a rotated extrinsic parameter matrix.

[0017] Furthermore, the specific implementation process of step S3 includes:

[0018] S31. Prediction of optical flow induced by vehicle rotation:

[0019] Convert the event point pixel coordinates to normalized camera coordinates, represented as:

[0020] and ;

[0021] in, and This indicates the focal length of the event camera in the horizontal and vertical directions. and Indicates the coordinates of the principal point of the event camera. and Represents normalized camera plane coordinates;

[0022] Based on the angular velocity in the event camera coordinate system obtained in step S2 The normalized optical flow induced by the rotation of the vehicle is calculated using the following formula:

[0023] ;

[0024] ;

[0025] in, and This represents the velocity in the normalized plane caused by the rotation of the vehicle. , and These represent the three components of the angular velocity in the event camera coordinate system;

[0026] Convert the normalized optical flow to the vehicle rotation-induced optical flow in pixel coordinates:

[0027] ;

[0028] ;

[0029] in, and This indicates that the rotation-induced optical flow at the location of the i-th event point is obtained by predicting the rotation of the vehicle.

[0030] S32. A compensation confidence level is introduced to weight the rotation-induced optical flow of the vehicle; the compensation confidence level is determined based on the continuity of IMU data, the rate of change of angular velocity, the acceleration modulus, and the consistency of background optical flow.

[0031] S33. Subtract the self-rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow.

[0032] Furthermore, the implementation method of S4 includes:

[0033] The optical flow points in the residual optical flow are screened, and the optical flow points that meet the following conditions are identified as candidate events for transverse sudden motion: the absolute value of the residual transverse velocity corresponding to the optical flow point is greater than the residual transverse velocity threshold, and the absolute value of the residual transverse velocity corresponding to the optical flow point is greater than the product of the absolute value of its residual longitudinal velocity and the preset transverse dominant ratio coefficient.

[0034] The number of candidate events was counted.

[0035] The residual transverse velocity amplitude is calculated based on the residual transverse velocity amplitude of the residual optical flow point corresponding to each candidate event;

[0036] Divide the image along the horizontal direction of the candidate event into multiple intervals, count the number of candidate events in each interval, and calculate the spatial clustering degree based on the number of candidate events in each interval.

[0037] The consistency of the lateral motion direction of the candidate events is obtained by statistically analyzing the residual lateral velocity sign of the residual optical flow points corresponding to each candidate event.

[0038] Furthermore, the implementation method of S5 includes:

[0039] Calculate the degree of deviation of the current window candidate event count relative to the historical window candidate event statistics to obtain the degree of time burst;

[0040] The weight of the danger zone is determined based on the region where the centroid of the candidate event is located.

[0041] The comprehensive risk score is obtained by weighting and integrating the number of candidate events after normalization, the horizontal strength of residuals, the degree of temporal burst, the spatial clustering, the directional consistency, and the weight of dangerous areas.

[0042] Furthermore, the risk level determination rule in step S6 is as follows:

[0043] ;

[0044] in, , and To preset risk thresholds, and < < .

[0045] A foreign object intrusion detection system based on event camera optical flow and IMU vehicle motion compensation includes: a front-end sensing layer, an edge computing processing layer, and an output and platform layer;

[0046] The front-end sensing layer includes an event camera acquisition module and an IMU acquisition module. The event camera acquisition module is used to acquire asynchronous event streams of road scenes; the IMU acquisition module is used to acquire IMU data, which includes vehicle angular velocity and acceleration data.

[0047] The edge computing processing layer includes a time synchronization module, an event stream preprocessing module, a sparse optical flow estimation module, a vehicle motion estimation module, a vehicle rotation-induced optical flow compensation module, a residual optical flow detection module, and a ghost carbon probe risk assessment module; wherein:

[0048] The time synchronization module receives the asynchronous event stream provided by the event camera acquisition module and the vehicle angular velocity and acceleration data from the IMU acquisition module, and aligns the event stream and IMU data according to the time window based on the timestamp;

[0049] The event stream preprocessing module receives the aligned event stream provided by the time synchronization module, and performs spatiotemporal denoising processing on the aligned event stream to obtain valid events;

[0050] The sparse optical flow estimation module receives valid events provided by the event stream preprocessing module and calculates the pixel motion velocity corresponding to each valid event to obtain a sparse observation optical flow set.

[0051] The vehicle motion estimation module receives aligned IMU data and event streams provided by the time synchronization module, and estimates the vehicle angular velocity in the event camera coordinate system for each time window based on the angular velocity and the external parameter relationship between the event camera and the IMU.

[0052] The vehicle rotation-induced optical flow compensation module receives the sparse observation optical flow set provided by the sparse optical flow estimation module and the vehicle angular velocity provided by the vehicle motion estimation module; based on the camera intrinsic parameters, event point coordinates and vehicle angular velocity, it predicts the image plane optical flow generated by the vehicle motion, denoted as the vehicle rotation-induced optical flow; and subtracts the vehicle rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow.

[0053] The residual optical flow detection module receives the residual optical flow provided by the vehicle rotation-induced optical flow compensation module, performs residual lateral velocity screening and lateral dominance judgment on the optical flow points in the residual optical flow, extracts candidate events for sudden lateral motion, and obtains the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters based on the candidate events.

[0054] The ghost carbon detector risk assessment module is used to: calculate the degree of temporal burst and the weight of dangerous areas; receive the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters provided by the residual optical flow detection module; calculate the comprehensive risk score based on the number of candidate events, residual lateral intensity, spatial clustering and directional consistency, degree of temporal burst and the weight of dangerous areas; and determine the candidate areas for ghost detectors based on the comprehensive risk score.

[0055] The output and platform layer includes a graded early warning module and a prompting module. The graded early warning module receives a comprehensive risk score from the ghost carbon detector risk assessment module, determines the risk level of candidate ghost detector areas based on the risk score, and sends the risk level determination result to the prompting module. The prompting module receives the risk level determination result from the graded early warning module and executes corresponding safety measures according to the risk level determination result, including audible and visual alarms, vehicle prompts, and active safety interfaces.

[0056] By adopting the above technical solution, the present invention has the following beneficial effects:

[0057] 1. The event camera asynchronously outputs timestamped events when pixel brightness changes, without waiting for a fixed frame period to complete a full frame exposure. When a pedestrian or electric vehicle suddenly enters laterally from an obstructed area, a set of events with dense temporal movement and obvious lateral motion is formed within a short time window. This invention estimates sparse observation optical flow based on this event set and completes the screening of candidate events for sudden lateral motion. Therefore, it can sense sudden lateral motion within a single time window and reduce the response delay caused by motion blur in frame-based imaging.

[0058] 2. This invention uses the angular velocity measured by the IMU to estimate the rotation-induced optical flow of the vehicle caused by turning, changing lanes or attitude disturbances, and introduces a compensation confidence level to weight the rotation-induced optical flow of the vehicle to compensate at the optical flow level, thereby reducing false alarms caused by the lateral optical flow of the static background.

[0059] 3. The sparse observation optical flow includes background motion components caused by vehicle rotation, independent target motion components, and noise components. This invention estimates the vehicle rotation-induced optical flow based on IMU-measured angular velocities and calculates the residual optical flow. The motion of the static background is mainly caused by vehicle rotation, and its residual approaches zero after compensation. Independent moving targets do not satisfy the vehicle rotation motion model, but retain a large residual lateral velocity after compensation, thus improving the salience of independent lateral moving targets relative to the background.

[0060] 4. The number of candidate events and the lateral intensity of the residuals are used to reflect the scale and intensity of sudden movements; spatial clustering and directional consistency are used to determine whether candidate events originate from local targets with consistent movement trends; the degree of temporal abruptness is used to distinguish between suddenly appearing targets and persistent background events; and the hazard zone weight is used to reflect the collision risk at the location of candidate events. Normalizing and weighting these factors can prevent a single factor from being directly triggered by noise, changes in illumination, or local background movements, thereby improving the stability of risk assessment.

[0061] 5. This invention can output graded early warning results, and is applicable to vehicle active safety prompts, platform recording, accident retrospection, and active safety interface linkage.

[0062] In summary, this invention solves the problems of existing ghost detection methods based on event camera optical flow being easily affected by background lateral optical flow interference when vehicles are turning, changing lanes, or experiencing vehicle attitude disturbances, thus leading to false alarms. Attached Figure Description

[0063] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0064] Figure 1 This is an overall flowchart of the method of the present invention;

[0065] Figure 2 A schematic diagram illustrating the fusion acquisition and time synchronization of event camera and IMU;

[0066] Figure 3 This is a schematic diagram of IMU-based optical flow compensation for vehicle rotation.

[0067] Figure 4 The flowchart shows the process for detecting lateral sudden motion based on residual optical flow.

[0068] Figure 5 This is a structural diagram of a risk scoring and grading early warning system for "ghost peeks". Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] like Figure 1 As shown, the foreign object intrusion detection method based on event camera optical flow and IMU vehicle motion compensation provided in this embodiment includes the following steps: multi-source data synchronous acquisition, event flow preprocessing and sparse optical flow estimation, IMU vehicle motion estimation, vehicle rotation-induced optical flow compensation, residual optical flow lateral detection, risk scoring, and graded early warning output. The specific execution process of each step is as follows:

[0071] S1. Multi-source data synchronous acquisition and alignment: An event camera is used to acquire asynchronous event streams ahead of the road, and an IMU is used to acquire IMU data; the acquired event streams and IMU data are time-synchronized, and the synchronized data is divided into time windows. Based on timestamps, the event streams and IMU data are aligned according to the time windows; the IMU data includes vehicle angular velocity and acceleration data; the implementation process is as follows: Figure 2 As shown, it specifically includes:

[0072] S11. Sensor Installation and Data Acquisition. In this embodiment, the event camera is installed on a rigid bracket inside the vehicle's windshield, in the center of the front of the vehicle, or on the front side. Its field of view covers the driving risk area in front of the vehicle, the entry areas on both sides of the road, and the roadside obstruction boundary area. The IMU is installed on a rigid part of the vehicle body or integrated with the event camera to collect the vehicle's angular velocity and acceleration. If the event camera and IMU are installed separately, the rotational extrinsic parameter matrix between the event camera and the IMU is obtained during the system calibration phase.

[0073] A single event captured by the event camera is represented as: ;in, Let x be the x-coordinate of the i-th time in the time camera pixel coordinate system. Let be the ordinate of the i-th event in the event camera pixel coordinate system. For event timestamps, The polarity of brightness change.

[0074] IMU measurement data is represented as follows: ;in, For the angular velocity measured by the j-th IMU, For the acceleration measured by the j-th IMU, This is the IMU timestamp.

[0075] S12. Time Window Division and Data Alignment. To achieve time alignment between the time stream and IMU data, this step divides the continuous event stream and measurement data into a unified time window. Specifically:

[0076] Based on time window length Divide the continuous event stream into several event windows, and define the set of events within the k-th time window. Represented as:

[0077] ;

[0078] in, This represents the set of events within the k-th time window. Indicates the start time of the window. Indicates the window length. Correspondingly, this relates to the event window. Time-aligned IMU datasets are represented as By ensuring that event data and IMU data fall within the same time interval. Thus establishing and The one-to-one correspondence between them, and the completion time alignment.

[0079] If the IMU sampling frequency is higher than the event window frequency, then the angular velocity within the window is averaged, filtered, or integrated; if the IMU sampling frequency is lower than the event window frequency, then the measurements of adjacent IMUs are interpolated so that each event window has a corresponding vehicle motion estimation input.

[0080] In this embodiment, the time window length is set to 33.3 ms. In practical applications, the time window length can also be dynamically adjusted according to vehicle speed, road complexity, or computing resources. When the vehicle speed is higher than the set threshold (which can be set manually; in this experiment, it is 10-35 km / h) or when the centroid of the candidate event is located in the lateral entry zone of the road, the window length is shortened to 80% of the original length to improve the response speed. When the number of events is less than the threshold (which was obtained in the test), the window length is appropriately increased to 120% of the original length to improve the stability of optical flow estimation.

[0081] S2. Event sparse optical flow and IMU angular velocity estimation, where:

[0082] S21. The specific process of estimating event sparse optical flow includes:

[0083] Spatiotemporal denoising is used to preprocess the event data stream to obtain valid events. The spatiotemporal denoising method in this embodiment is as follows: for the event window... any event in The system determines whether an event has supporting neighboring events within a preset spatial and temporal neighborhood (judgment method: for an event set, search for neighboring events with the same polarity within a spatial radius r and a time length τ, forming a neighborhood set N. When |N| is less than a fixed number, the event is considered isolated noise; otherwise, it is retained as a valid event). If the event lacks related events in its spatiotemporal neighborhood, or if its polarity change is inconsistent with surrounding events, it is considered an isolated noise event and discarded. For abnormal events caused by reflections, flickering light sources, raindrops, or sensor thermal noise, filtering is performed by combining local event density, polarity distribution, and duration.

[0084] Based on the effective event set, the event-level sparse optical flow is estimated, resulting in the observed optical flow set, denoted as: ,in, This represents the set of observed optical flow within the k-th time window. This represents the observation velocity of the i-th event point in the horizontal direction of the image. This represents the observation velocity of the i-th event point in the vertical direction of the image.

[0085] In engineering implementation, this step can also synchronously generate event frames for display and save sparse optical flow data. When playback or offline analysis is required, the sparse optical flow points in each window are saved in list form; when fusion with other sensing algorithms is required, the sparse optical flow can be aggregated by pixels or grid to generate a dense optical flow map, and the start and end timestamps of the corresponding windows are recorded. It should be noted that in subsequent processing, if there are no valid events or the number of valid observed optical flow points in the current window is insufficient, the risk score of the current window is skipped, or the risk value of the current window is set to a low-risk state.

[0086] S22. Vehicle angular velocity estimation in the event camera coordinate system: Vehicle turning, lane changing, or vehicle body shaking will cause global motion of the static background on the image plane, especially producing significant lateral optical flow. To suppress the interference caused by this type of background motion, this embodiment uses IMU data within the synchronization time window to estimate the vehicle angular velocity in the event camera coordinate system; the implementation process specifically includes the following sub-steps:

[0087] For IMU window The angular velocities within the vehicle are subjected to zero-bias correction and filtering to obtain estimated angular velocities in the vehicle coordinate system or IMU coordinate system. .

[0088] Based on the rotational extrinsic parameter relationship between the camera and the IMU coordinate system, the obtained angular velocity estimate is... Converted to angular velocity in the event camera coordinate system, it is expressed as:

[0089] ;

[0090] in, Represents the angular velocity in the event camera coordinate system. Represents the angular velocity in the vehicle coordinate system or IMU coordinate system. This represents the rotation extrinsic parameter matrix from the vehicle coordinate system or IMU coordinate system to the event camera coordinate system.

[0091] The angular velocity obtained in this step This will be used as input to step S4 for subsequent calculation of the image plane optical flow caused by the vehicle's rotational motion, denoted as the vehicle rotation-induced optical flow.

[0092] S3. Prediction of Vehicle Rotation-Induced Optical Flow and Calculation of Residual Optical Flow: Based on camera intrinsic parameters, event point coordinates, and vehicle angular velocity, predict the vehicle rotation-induced optical flow; subtract the vehicle rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow. The implementation process is as follows: Figure 3 As shown, the specific steps include:

[0093] S31. Prediction of optical flow induced by vehicle rotation, the specific implementation process is as follows:

[0094] Convert the event point pixel coordinates to normalized camera coordinates, represented as:

[0095] and ;

[0096] in, and This indicates the focal length of the event camera in the horizontal and vertical directions. and Indicates the coordinates of the principal point of the event camera. and Represents the normalized camera plane coordinates.

[0097] Based on the angular velocity in the event camera coordinate system output in step S2 The normalized optical flow induced by the rotation of the vehicle is calculated using the following formula:

[0098] ;

[0099] ;

[0100] in, and This represents the velocity in the normalized plane caused by the rotation of the vehicle. , and These represent the three components of the angular velocity in the event camera coordinate system. It should be noted that this implementation only considers the optical flow induced by the vehicle's rotation and does not consider the translation component for the following reasons: (1) In the scenario of the vehicle traveling at high speed, rotational motion (turning, changing lanes) is the main cause of the global motion of the background; (2) The amplitude of the optical flow caused by translation is coupled with the scene depth, and it is difficult to accurately estimate when the monocular event camera lacks depth information.

[0101] Convert the normalized optical flow to the vehicle rotation-induced optical flow in pixel coordinates:

[0102] ;

[0103] ;

[0104] in, and This indicates that the rotation-induced optical flow at the location of the i-th event point is obtained by predicting the rotation of the vehicle.

[0105] S32. Adaptive Compensation Weight Estimation:

[0106] Eigenvalues The characteristic of the optical flow being smaller when the vehicle is traveling straight and in a stable state, and larger when the vehicle is turning, changing lanes, or experiencing bumps, directly causes the rotation-induced optical flow of the vehicle to change accordingly, and its magnitude directly determines the amplitude of background motion compensation.

[0107] To avoid incorrectly compensating for the optical flow of real dynamic targets during IMU data anomalies or severe vehicle impacts, this embodiment utilizes eigenvalues. Based on the dynamic characteristics, an adaptive compensation weight is introduced. The weight estimation takes into account the following four factors:

[0108] IMU data continuity: Check whether there are dropped data frames or timestamp jumps within the IMU window. The better the continuity, the higher the weight.

[0109] Rate of change of angular velocity: Calculation The time derivative of the time derivative has a lower weight when the rate of change is too large, such as during a severe shock.

[0110] Acceleration Magnitude: When the acceleration magnitude significantly exceeds the normal driving range (such as during emergency braking or collision), it is assumed that the IMU may be affected by non-rotational motion interference, and its weight is reduced.

[0111] Background optical flow consistency: The observed optical flow and the predicted vehicle rotation-induced optical flow are matched for consistency across the entire map. For example, normalized cross-correlation is used. If the two trends are consistent in most background areas, the weight is increased; if the consistency is poor, it indicates that there are many independent moving targets or IMU estimation bias, and the weight is appropriately reduced.

[0112] Therefore, the weights are calculated. ;

[0113] Each sub-weight is obtained by mapping the confidence level of the corresponding indicator. The adjustable hyperparameter is, The function restricts the result to the interval [0,1].

[0114] S33. Residual Optical Flow Calculation:

[0115] Subtracting the rotation-induced optical flow from the observed optical flow yields the residual optical flow:

[0116] ;

[0117] ;

[0118] in, and This represents the compensated residual transverse optical flow and residual longitudinal optical flow. and This is the observed optical flow estimated from event data.

[0119] This step is a key step that distinguishes this invention from lateral detection methods based solely on the optical flow of the original event. When When the value is approximately 1, the IMU data is reliable and the background motion is consistent. For a static background, most of the motion in the observed optical flow can be explained by the optical flow induced by the vehicle's rotation, resulting in a small residual after compensation; when... When the value is approximately 0, indicating abnormal IMU data or a large number of independent moving targets in the scene, the system reduces or even cancels compensation to avoid falsely eliminating real dynamic targets. For pedestrians or electric vehicles that suddenly enter the road from the occlusion boundary, their motion does not satisfy the pure rotation model of the vehicle background motion model. Under arbitrary weight settings, a large residual lateral optical flow will still be retained after compensation, which is convenient for subsequent target detection.

[0120] S4. Lateral burst motion detection based on residual optical flow. (See also...) Figure 4 This step uses residual optical flow as input to sequentially complete lateral dominant candidate screening, candidate number statistics, spatial clustering analysis, and directional consistency analysis to obtain information related to ghost-protruding candidates. Specifically:

[0121] S41. After obtaining the residual optical flow, perform lateral dominance filtering on each optical flow point. The set of burst candidate events within the k-th window obtained by filtering is represented as:

[0122] ;

[0123] in, Let k be the set of horizontal burst candidate events within the k-th window. The residual lateral velocity threshold. This is the lateral dominance ratio coefficient. This screening condition is used to select event points from the residual optical flow that exhibit significant lateral velocity and whose lateral component is dominant.

[0124] S42. After filtering, count the number of candidate events. The number of candidate events is expressed as... ,in, This indicates the number of candidate events in the current window that meet the horizontal dominance condition. If If the number of ghost probes is below the preset minimum threshold, it is determined that the current window cannot form a stable candidate region for ghost probes.

[0125] S43. To distinguish between real local targets and global background disturbances, spatial clustering is calculated. The method for calculating spatial clustering is as follows:

[0126] The image corresponding to the residual optical flow is divided into multiple intervals along the horizontal direction. Count the number of candidate events within each interval:

[0127] ;

[0128] in, This represents the number of candidate events in the b-th horizontal interval.

[0129] Spatial clustering is obtained based on interval statistical results:

[0130] ;

[0131] in, This indicates the degree of local clustering of candidate events in the horizontal space. To prevent small constants with a denominator of zero. A higher value indicates that the candidate events are more concentrated in local areas, which is more consistent with the characteristics of pedestrians and electric vehicles suddenly appearing from the occlusion boundary.

[0132] After obtaining the spatial clustering degree, directional consistency is calculated, denoted as: ,

[0133] in, This indicates the degree of consistency in the lateral motion direction of candidate events, and the statistical basis is the sign of the residual lateral velocity of the residual optical flow point corresponding to each candidate event. If the candidate events belong to the same lateral entry into the target, the direction of the residual lateral velocity is usually relatively consistent; if it comes from noise or background disturbance, the consistency of direction is low.

[0134] S5. Ghost Pedaling Risk Scoring and Candidate Area Determination. (See also...) Figure 4This step calculates a comprehensive risk score by assessing the degree of temporal abruptness and the weight of hazardous areas, and combining this with parameters such as the number of candidate events, residual lateral strength, spatial clustering, and directional consistency. It then determines candidate areas for "ghost peeks" (protruding individuals). Specifically:

[0135] To avoid using only a single threshold for alarms, each risk factor is solved sequentially, specifically:

[0136] Sudden increase in computation time:

[0137] ;

[0138] in, This represents the average number of candidate events in the historical window. This represents the standard deviation of the number of candidate events in the historical window. The degree of temporal burst measures the degree of burst in the number of candidate events in the current window relative to the historical background, highlighting the "sudden appearance" characteristic of a ghost-like target.

[0139] Calculate the residual lateral strength:

[0140] ;

[0141] in, This represents the magnitude of the average residual lateral velocity of the candidate event set; The larger the value, the more obvious the lateral movement of the candidate target into the vehicle's risk zone.

[0142] In this embodiment, the weight of the danger zone is determined based on the centroid of the candidate events. The region is assigned a value. The centroid of a candidate event is obtained by averaging the pixel coordinates corresponding to the candidate event. The detection region is divided into the vehicle front risk zone, the road side entry zone, the occlusion boundary neighborhood, and other regions. The region weighting rules are as follows:

[0143] ;

[0144] in, Indicates the risk zone ahead of the vehicle. Indicates the lateral access area of ​​the road. Indicates the occlusion boundary neighborhood. This represents other regions. The weight of this region is used to reflect the difference in collision risk caused by the location of the candidate event.

[0145] A comprehensive risk score is obtained by integrating all factors:

[0146] ;

[0147] in, Indicates the first Risk score for ghost protrusions within a time window; , and These represent the normalization functions corresponding to the number of candidates, the transverse strength of the residuals, and the degree of temporal burst, respectively. to This represents the weighting parameters for each risk factor.

[0148] S6. Tiered Early Warning and Result Output. (See also...) Figure 4 This step outputs a risk level based on the risk score obtained in step S5, and uses a tiered early warning module and a data recording and uploading module to achieve vehicle-mounted alerts, alarm linkage, and platform record keeping. Specifically:

[0149] Based on risk score Conduct a risk level assessment. The risk level assessment is defined as follows:

[0150] ;

[0151] in, , and To preset risk thresholds, and < < .

[0152] For Level I risks, the system triggers a strong alert, including audible and visual alarms, a red warning displayed on the vehicle screen, real-time push notifications from the platform, or output from the active safety interface; for Level II risks, the system triggers vehicle-mounted alerts and visual alarms from the management terminal; for Level III risks, the system records the information in the background and can display it during inspections or playback; for no-risk situations, the system continues to monitor without triggering external alarms.

[0153] Regarding result recording, the system can save event windows, sparse optical flow, dense optical flow, residual optical flow, risk scores, risk levels, alarm times, and window start and end timestamps. For Level I risks, it can save the event stream and visualization results for several seconds before and after the alarm for incident debriefing; for Level II risks, it can save the optical flow and scoring results of the critical window; for Level III risks, it can save only the risk summary.

[0154] The above-mentioned foreign object intrusion detection methods based on event camera optical flow and IMU vehicle motion compensation are summarized into a vehicle motion compensation algorithm based on event optical flow, and its pseudocode is as follows:

[0155] Among them, sparse flow is used to save event-level optical flow data, while dense flow is used to save optical flow fields aggregated by pixels or grids, and can be accompanied by window start and end timestamps to facilitate subsequent playback, calibration and model updates.

[0156] like Figure 5 As shown, based on the above detection method, this embodiment also provides a foreign object intrusion detection system based on event camera optical flow and IMU vehicle motion compensation. The system includes: a front-end sensing layer, an edge computing processing layer, and an output and platform layer.

[0157] The front-end sensing layer includes an event camera acquisition module and an IMU acquisition module. The event camera acquisition module is used to acquire asynchronous event streams of road scenes; the IMU acquisition module is used to acquire IMU data, which includes vehicle angular velocity and acceleration data.

[0158] The edge computing processing layer includes a time synchronization module, an event stream preprocessing module, a sparse optical flow estimation module, a vehicle motion estimation module, a vehicle rotation-induced optical flow compensation module, a residual optical flow detection module, and a ghost carbon probe risk assessment module; wherein:

[0159] The time synchronization module receives the asynchronous event stream provided by the event camera acquisition module and the vehicle angular velocity and acceleration data from the IMU acquisition module, and aligns the event stream and IMU data according to the time window based on the timestamp;

[0160] The event stream preprocessing module receives the aligned event stream provided by the time synchronization module, and performs spatiotemporal denoising processing on the aligned event stream to obtain valid events;

[0161] The sparse optical flow estimation module receives valid events provided by the event stream preprocessing module and calculates the pixel motion velocity corresponding to each valid event to obtain a sparse observation optical flow set.

[0162] The vehicle motion estimation module receives aligned IMU data and event streams provided by the time synchronization module, and estimates the vehicle angular velocity in the event camera coordinate system for each time window based on the angular velocity and the external parameter relationship between the event camera and the IMU.

[0163] The vehicle rotation-induced optical flow compensation module receives the sparse observation optical flow set provided by the sparse optical flow estimation module and the vehicle angular velocity provided by the vehicle motion estimation module; based on the camera intrinsic parameters, event point coordinates and vehicle angular velocity, it predicts the image plane optical flow generated by the vehicle motion, denoted as the vehicle rotation-induced optical flow; and subtracts the vehicle rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow.

[0164] The residual optical flow detection module receives the residual optical flow provided by the vehicle rotation-induced optical flow compensation module, performs residual lateral velocity screening and lateral dominance judgment on the optical flow points in the residual optical flow, extracts candidate events for sudden lateral motion, and obtains the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters based on the candidate events.

[0165] The ghost carbon detector risk assessment module receives the number of candidate events, residual lateral intensity, spatial clustering, and directional consistency parameters provided by the residual optical flow detection module; it calculates the time burst degree and the danger zone weight based on the number of candidate events, residual lateral intensity, spatial clustering, and directional consistency parameters to calculate a comprehensive risk score; and it determines the ghost detector candidate area based on the comprehensive risk score.

[0166] The output and platform layer includes a hierarchical early warning module, a notification module, and a data recording and uploading module; wherein:

[0167] The graded early warning module receives the comprehensive risk score provided by the ghost carbon detector risk assessment module, judges the risk level of the candidate area of ​​the ghost carbon detector based on the risk score, and sends the risk level judgment result to the prompt module.

[0168] The alert module receives the risk level assessment result provided by the graded warning module, and triggers corresponding audible and visual alarms, vehicle alerts, and active safety interfaces to provide warnings based on the risk level assessment result; it generates a warning log based on the warning data and sends it to the data recording and uploading module for storage and analysis; the warning log includes risk score, risk level, and timestamp, etc.

[0169] This embodiment also provides an electronic device, which may be an in-vehicle domain controller, an edge computing box, an industrial control computer, or an embedded processing unit, and includes at least a processor and a memory. The memory stores an executable program, and when the processor executes the program, it implements the aforementioned detection method. The electronic device may also include a communication interface, a video output interface, and an alarm control interface to enable linkage with an in-vehicle display screen, an audible and visual alarm, or an active safety system.

[0170] This embodiment also provides a computer-readable storage medium, which may be a read-only memory, flash memory, hard disk, solid-state drive, memory card, or other non-volatile storage medium, on which an executable program is stored. When the program is executed by a processor, it implements the aforementioned detection method, including event synchronization, noise reduction, sparse optical flow estimation, vehicle motion compensation, risk scoring, and graded early warning.

[0171] Deployment plan and experimental verification:

[0172] The detection method and system of this embodiment can be deployed in vehicle-mounted front-end perception devices, ADAS domain controllers, or edge computing platforms, adopting a three-layer architecture of "front-end acquisition - edge computing - central platform". The front end completes event camera and IMU acquisition, the edge side completes synchronization, denoising, optical flow estimation, residual compensation, and risk scoring, and the central platform is used for result archiving, model updating, and accident backtracking.

[0173] Deployment verification can be set up in the following scenarios: the vehicle is traveling straight and a pedestrian crosses from the right-side obstruction area; the vehicle is traveling straight and an electric vehicle exits from the left-side intersection; the vehicle turns left and there is no target crossing; the vehicle turns right and a pedestrian appears from behind a parked vehicle; the vehicle changes lanes and there is no sudden target ahead; sudden changes in lighting or low light conditions in rainy weather; and scenarios where pedestrians walk parallel to the roadside but do not enter the risk zone. Comparison schemes can include the original event optical flow detection scheme without IMU compensation, the scheme using IMU compensation but without spatial clustering, the scheme using IMU compensation but without temporal abrupt changes in features, and the complete scheme of this invention.

[0174] Verification metrics may include false alarm rate, false negative rate, average warning lead time, single-window processing latency, false alarm reduction rate in turning scenarios, and event processing throughput per unit time. In an exemplary verification scheme, the event camera resolution is 640×480, the event window length is 33.3ms, the IMU sampling frequency is 200Hz, the test vehicle speed is 10km / h to 35km / h, the average single-window processing latency of the edge computing platform is 24.6ms, and the 95th percentile processing latency is 31.8ms, meeting the real-time warning requirements. The above parameters are only the test conditions for this embodiment and do not constitute a limitation on the scope of protection of this invention.

[0175] Under the above test conditions, the following exemplary verification results can be obtained as shown in Table 1:

[0176] Table 1. Performance evaluation results of the present invention in different traffic scenarios for detecting ghost pedestrians:

[0177]

[0178] For vehicle motion scenarios where there are no real ghost-protruding targets but false alarms are common, a further comparison was made between the original event optical flow detection scheme and the complete scheme of this invention, yielding the following exemplary results, as shown in Table 2:

[0179] Table 2. Statistics of false alarm detection indicators for the present invention and the comparative scheme in various test scenarios:

[0180]

[0181] The above results show that in straight-line ghost-peek scenarios, the present invention maintains the event camera's rapid response capability to high-speed sudden targets; in targetless scenarios such as turning, lane changing, and attitude disturbances, the present invention significantly reduces false alarms caused by background lateral optical flow through IMU vehicle motion compensation; in occluded boundary scenarios, the residual optical flow can still retain the lateral sudden characteristics of independent moving targets, and improve the stability of risk scoring through spatial clustering, directional consistency, and temporal burst characteristics.

[0182] The above embodiments are merely illustrative of the principles and effects of the present invention and are not intended to limit the invention. Any person skilled in the art can modify or alter the above embodiments without departing from the spirit and scope of the present invention. Therefore, all equivalent modifications or alterations made by those skilled in the art without departing from the spirit and technical concept disclosed in the present invention should still be covered by the claims of the present invention.

Claims

1. A foreign object intrusion detection method based on event camera optical flow and IMU vehicle motion compensation, characterized in that, Includes the following steps: S1. Multi-source data synchronous acquisition and alignment: An event camera is used to acquire asynchronous event streams ahead of the road, and an IMU is used to acquire IMU data; the acquired event streams and IMU data are time-synchronized, and the synchronized data is divided into time windows. The event streams and IMU data are aligned according to the time windows based on the timestamps; the IMU data includes vehicle angular velocity and acceleration data; S2. Event Sparse Optical Flow and IMU Angular Velocity Estimation: Spatiotemporal denoising is performed on the aligned event flow to obtain valid events. The pixel motion velocity corresponding to each event is calculated based on the valid events to obtain a sparse observation optical flow set. Based on the aligned IMU angular velocities within each time window and the external parameter relationship between the event camera and the IMU, estimate the angular velocity of the vehicle in the event camera coordinate system within the current time window. S3. Prediction of vehicle rotation-induced optical flow and calculation of residual optical flow: Based on the camera intrinsic parameters, event point coordinates and vehicle angular velocity, predict the image plane optical flow generated by the vehicle motion, denoted as vehicle rotation-induced optical flow; subtract the vehicle rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow; S4. Lateral sudden motion detection based on residual optical flow: Perform residual lateral velocity screening and lateral dominance judgment on optical flow points in residual optical flow, extract lateral sudden motion candidate events, and obtain the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters based on the sudden motion candidate events. S5. Ghost Probe Risk Scoring and Candidate Area Determination: Calculate the degree of time surge and the weight of dangerous areas. Based on this, combine the number of candidate events, residual lateral strength, spatial clustering and directional consistency parameters to calculate the comprehensive risk score. Determine the ghost probe candidate area based on the comprehensive risk score. S6. Tiered Early Warning and Result Output: Based on the risk score, the risk level is determined according to the preset judgment rules, and corresponding safety measures are implemented based on the risk level determination results, including audible and visual alarms, vehicle prompts, platform recording, or active safety interfaces.

2. The method as described in claim 1, characterized in that, In step S1, the event stream and IMU data are synchronized using hardware timestamps or software interpolation. The synchronized event stream is then divided into a set of time windows with a preset duration. The length of the time window is a fixed value or can be adaptively adjusted according to the vehicle speed.

3. The method as described in claim 1, characterized in that, In step S2, the spatiotemporal denoising process for the aligned event stream includes: removing isolated events, flickering noise events, and abnormal events that do not satisfy neighborhood consistency.

4. The method as described in claim 1, characterized in that, The extrinsic parameter relationships in step S2 are obtained through pre-calibration, and the extrinsic parameter relationships are represented by a rotated extrinsic parameter matrix.

5. The method as described in claim 1, characterized in that, The specific implementation process of step S3 includes: S31. Prediction of optical flow induced by vehicle rotation: Convert the event point pixel coordinates to normalized camera coordinates, represented as: and ; in, and This indicates the focal length of the event camera in the horizontal and vertical directions. and Indicates the coordinates of the principal point of the event camera. and Represents normalized camera plane coordinates; Based on the angular velocity in the event camera coordinate system obtained in step S2 The normalized optical flow induced by the rotation of the vehicle is calculated using the following formula: ； ； in, and This represents the velocity in the normalized plane caused by the rotation of the vehicle. , and These represent the three components of the angular velocity in the event camera coordinate system; Convert the normalized optical flow to the vehicle rotation-induced optical flow in pixel coordinates: ； ； in, and This indicates that the rotation-induced optical flow at the location of the i-th event point is obtained by predicting the rotation of the vehicle. S32. A compensation confidence level is introduced to weight the rotation-induced optical flow of the vehicle; the compensation confidence level is determined based on the continuity of IMU data, the rate of change of angular velocity, the acceleration modulus, and the consistency of background optical flow. S33. Subtract the self-rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow.

6. The method according to claim 1, characterized in that, The implementation method of S4 includes: The optical flow points in the residual optical flow are screened, and the optical flow points that meet the following conditions are identified as candidate events for transverse sudden motion: the absolute value of the residual transverse velocity corresponding to the optical flow point is greater than the residual transverse velocity threshold, and the absolute value of the residual transverse velocity corresponding to the optical flow point is greater than the product of the absolute value of its residual longitudinal velocity and the preset transverse dominant ratio coefficient. The number of candidate events was counted. The residual transverse velocity amplitude is calculated based on the residual transverse velocity amplitude of the residual optical flow point corresponding to each candidate event; Divide the image along the horizontal direction of the candidate event into multiple intervals, count the number of candidate events in each interval, and calculate the spatial clustering degree based on the number of candidate events in each interval. The consistency of the lateral motion direction of the candidate events is obtained by statistically analyzing the residual lateral velocity sign of the residual optical flow points corresponding to each candidate event.

7. The method according to claim 1, characterized in that, The implementation method of S5 includes: Calculate the degree of deviation of the current window candidate event count relative to the historical window candidate event statistics to obtain the degree of time burst; The weight of the danger zone is determined based on the region where the centroid of the candidate event is located. The comprehensive risk score is obtained by weighting and integrating the number of candidate events after normalization, the horizontal strength of residuals, the degree of temporal burst, the spatial clustering, the directional consistency, and the weight of dangerous areas.

8. The method according to claim 1, characterized in that, The risk level determination rule in step S6 is as follows: ； in, , and To preset risk thresholds, and < < .

9. A foreign object intrusion detection system based on event camera optical flow and IMU vehicle motion compensation, characterized in that, include: Front-end sensing layer, edge computing processing layer, and output and platform layer; The front-end sensing layer includes an event camera acquisition module and an IMU acquisition module. The event camera acquisition module is used to acquire asynchronous event streams of road scenes; the IMU acquisition module is used to acquire IMU data, which includes vehicle angular velocity and acceleration data. The edge computing processing layer includes a time synchronization module, an event stream preprocessing module, a sparse optical flow estimation module, a vehicle motion estimation module, a vehicle rotation-induced optical flow compensation module, a residual optical flow detection module, and a ghost carbon probe risk assessment module; wherein: The time synchronization module receives the asynchronous event stream provided by the event camera acquisition module and the vehicle angular velocity and acceleration data from the IMU acquisition module, and aligns the event stream and IMU data according to the time window based on the timestamp; The event stream preprocessing module receives the aligned event stream provided by the time synchronization module, and performs spatiotemporal denoising processing on the aligned event stream to obtain valid events; The sparse optical flow estimation module receives valid events provided by the event stream preprocessing module and calculates the pixel motion velocity corresponding to each valid event to obtain a sparse observation optical flow set. The vehicle motion estimation module receives aligned IMU data and event streams provided by the time synchronization module, and estimates the vehicle angular velocity in the event camera coordinate system for each time window based on the angular velocity and the external parameter relationship between the event camera and the IMU. The vehicle rotation-induced optical flow compensation module receives the sparse observation optical flow set provided by the sparse optical flow estimation module and the vehicle angular velocity provided by the vehicle motion estimation module; based on the camera intrinsic parameters, event point coordinates and vehicle angular velocity, it predicts the image plane optical flow generated by the vehicle motion, denoted as the vehicle rotation-induced optical flow; and subtracts the vehicle rotation-induced optical flow from the sparse observation optical flow set to obtain the residual optical flow. The residual optical flow detection module receives the residual optical flow provided by the vehicle rotation-induced optical flow compensation module, performs residual lateral velocity screening and lateral dominance judgment on the optical flow points in the residual optical flow, extracts candidate events for sudden lateral motion, and obtains the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters based on the candidate events. The ghost carbon detector risk assessment module is used to: calculate the degree of temporal burst and the weight of dangerous areas; receive the number of candidate events, residual lateral intensity, spatial clustering and directional consistency parameters provided by the residual optical flow detection module; calculate the comprehensive risk score based on the number of candidate events, residual lateral intensity, spatial clustering and directional consistency, degree of temporal burst and the weight of dangerous areas; and determine the candidate areas for ghost detectors based on the comprehensive risk score. The output and platform layer includes a graded early warning module and a prompting module. The graded early warning module receives a comprehensive risk score from the ghost carbon detector risk assessment module, determines the risk level of candidate ghost detector areas based on the risk score, and sends the risk level determination result to the prompting module. The prompting module receives the risk level determination result from the graded early warning module and executes corresponding safety measures according to the risk level determination result, including audible and visual alarms, vehicle prompts, and active safety interfaces.

Images