Unmanned system multi-target tracking algorithm based on event driving, electronic equipment and readable storage medium

Abstract

The invention belongs to the technical field of computer vision and unmanned system perception, and particularly discloses an unmanned system multi-target tracking algorithm based on event driving, electronic equipment and a readable storage medium, and the method comprises the steps: S1, event data preprocessing: obtaining an asynchronous event stream output by an event camera, preprocessing the asynchronous event stream, and generating an event block; s2, multi-target initial feature extraction: extracting static features and dynamic features of each target; s3, event-driven tracking triggering: the obtained event density is used for triggering tracking updating of the target; and S4, multi-target association and conflict resolution: extracting real-time static features and real-time dynamic features of the potential target, and judging a matching state for updating and outputting a tracking state of the target.

Description

TECHNICAL FIELD

[0001] The present application belongs to the field of computer vision and unmanned system perception technology, and particularly relates to an event-driven unmanned system multi-target tracking algorithm, an electronic device and a readable storage medium. BACKGROUND

[0002] Multi-target tracking is a core technology module of the perception layer of an unmanned system, and its function is to continuously identify and output the position, speed and trajectory information of various traffic participants in a road scene, directly determining the response timeliness and control accuracy of the decision system (such as emergency braking and turning to avoid) of the unmanned system.

[0003] According to industry statistics, 70% of the perception layer failures and multi-target tracking delays (> 50ms) or ID confusion in unmanned system testing and landing cases, and the risk of rear-end collision caused by tracking lag in high-speed scenes is more than 3 times higher than in low-speed scenes, which is a key bottleneck restricting the landing of unmanned systems at L3 and above levels.

[0004] Existing multi-target tracking technologies for unmanned systems mainly rely on traditional frame exposure cameras (RGB cameras) or laser radar fusion solutions, but they have significant technical limitations in complex real-world scenarios:

[0005] ①Real-time performance cannot meet high-speed requirements: Since traditional frame cameras output images at a fixed frame rate (30-60fps), single-frame feature extraction and matching takes about 15-30ms, and after adding data transmission delay, the total response time exceeds 50ms; when a vehicle travels at a speed of 120km / h, a 50ms delay will cause the vehicle to travel an additional 1.67m, which is likely to miss the emergency obstacle avoidance window, and thus cause safety accidents.

[0006] ②Poor adaptability of anti-interference capability: Since the dynamic range of traditional frame cameras is only 60-80dB, it cannot cover the 120dB brightness difference in the "strong light reflection area (such as road water reflection)-shadow area (such as under a bridge)" of the unmanned system scene, and overexposure or underexposure will cause target feature loss, resulting in a small vehicle (such as a motorcycle) missing rate exceeding 25% and a pedestrian crossing scene misjudgment rate exceeding 30%; at the same time, environmental disturbances such as raindrops and snowflakes will further exacerbate the distortion of frame image features, affecting tracking stability.

[0007] ③Data redundancy consumes computing resources: Since traditional frame cameras output complete images regardless of whether the scene changes or not, and the static background (such as the road surface and green belt) accounts for more than 80%, it occupies more than 40% of the computing power of the vehicle-mounted chip (such as GPU) without real-time processing, resulting in a competition for resources between the multi-target tracking module and other perception functions (such as semantic segmentation and lane detection), reducing the overall system running efficiency.

[0008] As a new dynamic vision sensor, the event camera (Dynamic Vision Sensor, DVS) only outputs an asynchronous event stream (containing spatial coordinates (x, y), timestamp t, and gray scale change polarity p) when the gray scale value of the pixel changes by a preset threshold, has a microsecond-level time resolution (<1us), a high dynamic range (>=120dB), no motion blur, and low data redundancy (single frame equivalent data volume is only 1 / 10 of that of the frame camera), etc. Advantages provide a new path for solving the real-time and anti-interference problems of multi-target tracking of unmanned systems.

[0009] However, the existing multi-target tracking technology based on event cameras still has key technical problems:

[0010] ① Insufficient scene adaptability: The existing multi-target tracking technology based on event cameras mostly uses a fixed time window (such as 1ms) to group event streams, and does not dynamically adjust according to the target motion speed. In high-speed vehicle scenes, the motion details are lost due to the large window, and in low-speed pedestrian scenes, feature fragmentation occurs due to the small window, which cannot meet the tracking needs of different traffic participants;

[0011] ② Low robustness of trigger mechanism: The existing multi-target tracking technology based on event cameras mostly relies on the number of events (such as single-window event number >100) to trigger tracking updates, and does not set conditions in combination with target motion states (such as sudden acceleration, turning), which cannot achieve millisecond-level response in dangerous scenes such as vehicle sudden braking (speed change rate >25%) and pedestrian sudden crossing (direction angle change >15°);

[0012] ③ Unreasonable feature fusion logic: The existing multi-target tracking technology based on event cameras treats static features (such as contour Hu moment) and dynamic features (such as motion speed) equally, and does not consider the core demand of "motion state is more critical than static contour" in unmanned systems, resulting in an ID confusion rate of more than 20% in vehicle merging scenes;

[0013] ④ Weak conflict resolution capability: The existing multi-target tracking technology based on event cameras lacks effective position prediction and re-matching mechanism in the face of scenes such as large vehicle occluding small vehicle (IOU >0.5) and target temporary loss (no matching for 3 consecutive windows), and the tracking recovery rate is less than 60%, which is difficult to support continuous and stable perception output. SUMMARY

[0014] In order to solve the above technical problems, the present application provides an event-driven multi-target tracking algorithm for unmanned systems, an electronic device and a readable storage medium.

[0015] Based on the above purpose, the present application is realized by the following technical scheme:

[0016] The first aspect of this invention provides an event-driven multi-target tracking algorithm for unmanned systems, comprising the following steps:

[0017] S1. Event Data Preprocessing: Obtain the asynchronous event stream output by the event camera, preprocess the asynchronous event stream, and generate event blocks.

[0018] S2. Initial feature extraction for multiple targets: Extract the static and dynamic features of each target.

[0019] S3, Event-Driven Tracking Trigger: The obtained event density is used to trigger tracking updates for the target.

[0020] S4. Multi-target association and conflict resolution: Extract real-time static and dynamic features of potential targets, determine the matching status, and use them for target tracking status updates and output.

[0021] According to the above event-driven multi-target tracking algorithm for unmanned systems, preferably, in step S1, the asynchronous event stream is collected by the event camera mounted on the unmanned vehicle, and each event is represented as:

[0022] (x,y,t,p);

[0023] Where x and y are pixel coordinates, t is a timestamp, and p∈{+1,-1} is the direction of grayscale change (reflecting the brightness changes between traffic participants and the background).

[0024] The asynchronous event stream is then denoised to filter out isolated noise events caused by road surface reflections, sensor jitter, or raindrop interference, as shown below:

[0025] For noise generated by strong light, shadow or road bumps in unmanned system scenarios, for an event (x,y,t,p), if the number of events in its 3×3 neighborhood that satisfy |t′-t|≤τ is less than N0, it is judged as a noise event and filtered out.

[0026] N0=ρ min ·9·τ;

[0027] Where N0 is the quantity threshold (3-5); ρ min τ represents the minimum event density for the scene, with a value of 0.5 events per (pixel·ms), corresponding to the event generation rate of slow-moving pedestrians; τ is the time threshold (0.1ms-0.5ms, adapted to the time resolution of high-speed moving targets).

[0028] The denoised event stream is grouped to generate event block E. k :

[0029] E k ={(x i ,y i,t i ,p i |t∈[kΔt,(k+1)Δt)};

[0030] Where k is the event block index; Δt is the preset time window, which adapts to the movement speed of the target in the unmanned system scenario (e.g., a smaller Δt is needed for high-speed vehicles to avoid information loss).

[0031] The preset time window Δt satisfies Δt=max(0.5ms,1 / ρ avg );

[0032] Where, ρ avg The average event density in unmanned system scenarios is ρ, which is higher on highways due to the rapid movement of vehicles. avg Higher, Δt smaller.

[0033] The event block E k The minimum number of events is:

[0034]

[0035] Among them, S min Minimum target area, unit: pixel 2 Pixel size appropriate for children or small vehicles.

[0036] Based on the above event-driven multi-target tracking algorithm for unmanned systems, preferably, in step S2, for the first valid event block E0 in the event block, N targets of traffic participants to be tracked (N≥2, such as multiple cars and pedestrians at an intersection) are identified through connected component analysis, and each target corresponds to a region. For each target n, extract static and dynamic features.

[0037] The static features include bounding box parameter B used to adapt to different contour sizes of vehicles and pedestrians. n =(x min ,x max ,y min ,y max ) and contour Hu moment h n =(h n1 ,h n2 ,...,h n7 ).

[0038] h n1 ,h n2 ,...,h n7 Represented as h ni h ni Let Hu be the i-th order moment, which is expressed as:

[0039] h ni =∫∫(x,y)i I(x,y)dxdy;

[0040] Where I(x,y) is the event density function of event block E0 at (x,y), that is, the number of events per unit area, used to distinguish between vehicles (high-density area) and pedestrians (medium-density area). min The minimum value of the x-coordinate; x max The maximum value of the x-coordinate; y min The minimum value of the y-coordinate; y max y is the maximum value of the y-coordinate, corresponding to the upper and lower boundaries of the integration region in the x and y directions; dx and dy are the differential elements in the x and y directions, respectively, used to divide the two-dimensional region into infinitesimal elements.

[0041] The dynamic features include a histogram D representing the direction of grayscale changes that reflects the movement of the target surface texture (such as the difference between a vehicle body and a pedestrian's clothing). n and initial velocity

[0042] Represented as:

[0043]

[0044] in, For R n The center coordinates; For R n Average timestamp of internal events; |R n |for R n The number of events within the system, speed calculation adapted to the road coordinate system; v nx Corresponding to lateral motion, v ny Corresponding to longitudinal movement.

[0045] Based on the above event-driven multi-target tracking algorithm for unmanned systems, preferably, in step S3, the subsequent event block E is... k Real-time calculation of each target n in the current region Event density within;

[0046]

[0047] Wherein, n is the target in the k-th event block region. Event density;

[0048]

[0049] in, Let be the change in the direction angle of motion of target n at time k and time k-1; This is the current direction angle, reflecting vehicle turning or pedestrian changing direction; Let be the direction angle of motion of target n at time k-1.

[0050]

[0051] in, Let n be the rate of change of relative velocity of target n at time k, reflecting rapid acceleration / deceleration; Let n be the velocity vector of target n at time k-1.

[0052] exist or or At that time, the tracking update of target n is triggered to quickly respond to potential dangers in unmanned systems.

[0053] Wherein, ρ0 is the first threshold, corresponding to the target entering the near field or undergoing violent movement; θ0 is the second threshold, corresponding to vehicles turning or pedestrians crossing the road; α0 is the third threshold, corresponding to emergency braking or acceleration.

[0054] Based on the above event-driven multi-target tracking algorithm for unmanned systems, preferably, in step S4, the event block E that triggers the update is updated. k Extract real-time static features of potential target m and real-time dynamic features Calculate the feature similarity S(n,m) with the historical target n:

[0055] S(n,m)=w s ·S s (n,m)+w d ·S d (n,m);

[0056] Among them, w s +w d =1, w s For static feature weights, w d For dynamic feature weights, dynamic weights adapt to the priority of target motion in unmanned systems; S s (n,m) represents the static feature similarity; S d (n,m) represents the dynamic feature similarity.

[0057] Static feature similarity is represented as:

[0058]

[0059] Where, ||·|2 is the Euclidean distance, used to match the contour consistency of vehicles / pedestrians; h n , These are the static feature vectors of targets n and m, respectively, with superscripts used to distinguish them from h. n Perform a similarity comparison;

[0060] Dynamic feature similarity is represented as:

[0061]

[0062] in,‖·‖ B D is the Barthelium distance, β is an adjustment parameter, β = 0.8, used to match motion patterns, such as the difference between a vehicle moving at a constant speed and a pedestrian moving at varying speeds; n , Let D represent the dynamic feature distributions of targets n and m, respectively. The superscript here is used to distinguish them from D. n Perform a similarity comparison;

[0063] An association matrix is ​​constructed based on feature similarity S(n,m), and the optimal match is solved using the Hungarian algorithm to assign historical target IDs to potential targets. If there are unmatched targets with S(n,m) < γ0 (such as pedestrians suddenly appearing at intersections), the historical target positions are predicted using Kalman filtering. (Adapted to road planar motion model), represented as:

[0064]

[0065] Where F is the state transition matrix and G is the input matrix.

[0066] Recalculate the Intersection over Union (IOU) ratio between the predicted location and the potential target to resolve ID conflicts (such as target confusion during vehicle merging).

[0067] Tracking status updates and output: Update the real-time positions of each target based on the correlation results. The road location mapped to the world coordinate system, and the trajectory used to predict the target driving path. Tracking status; output historical target IDs and historical target locations for all targets. speed Provide obstacle dynamic information for the decision-making system in unmanned systems.

[0068] Based on the above event-driven multi-target tracking algorithm for unmanned systems, preferably, the state transition matrix of the Kalman filter is:

[0069]

[0070] Its input matrix is:

[0071]

[0072] Its state vector is:

[0073] pos n =(p,q,v) x ,vy ) T ;

[0074] Where p, q are the positions in the road coordinate system; v x ,v y It provides lateral and longitudinal velocities, adapting to the planar motion models of vehicles and pedestrians.

[0075] According to the above event-driven multi-target tracking algorithm for unmanned systems, preferably, the tracking states include normal tracking / occlusion tracking / loss tracking;

[0076] The criteria for determining occlusion tracking are:

[0077] Intersection over Union (IOU) of the regions of target n and target m n ,R m When ) > 0.5 (e.g., a large vehicle obstructing a small vehicle), use the expression. Update the predicted location.

[0078] The criteria for determining "tracking loss" are:

[0079] If K consecutive event blocks fail to match (K=3), trigger a re-detection: for event block E k For events in the dataset that have not been assigned a historical target ID (which may be recurring targets), a candidate region C is generated, represented as:

[0080]

[0081] Where (p,q) are the spatial coordinates of the candidate region, (p',q') are the spatial coordinates of the event in the event block, t is the temporal attribute of the event, p is other attributes of the event, λ is the spatial decay factor, and η is the candidate region threshold. It adapts to the target distribution range of the road scene and matches the features of historically lost targets (such as vehicles that reappear after being occluded).

[0082] Based on the aforementioned event-driven multi-target tracking algorithm for unmanned systems, preferably, dynamic feature optimization is performed on the high-speed motion characteristics of the vehicle target in the unmanned system: the initial motion speed... Replace with event-stream based optical flow vectors To more accurately reflect the relative motion of the road surface.

[0083] Optical flow vector Represented as:

[0084]

[0085] Where, x t+δ y t+δLet x and y be the spatial coordinates of the event after a time interval δ, t be the temporal attribute of the event, and p be other attributes of the event R. n The target area is defined by δ, which represents the optical flow calculation time interval, adapting to the instantaneous speed capture of vehicles moving at high speeds.

[0086] A second aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step in the event-driven multi-target tracking algorithm for unmanned systems as described in the first aspect.

[0087] A third aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a computer processor, implements any step in the event-driven multi-target tracking algorithm for unmanned systems as described in the first aspect.

[0088] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0089] 1. This invention, by combining scene adaptive parameter adjustment, motion state triggering, dynamic feature priority fusion, and intelligent conflict, can solve the technical problems of insufficient real-time performance and susceptibility to high-speed movement, target occlusion, and environmental noise when existing algorithms track multiple traffic participants such as vehicles, pedestrians, and cyclists in complex autonomous driving scenarios. It can improve the driving safety and environmental perception reliability of unmanned vehicles.

[0090] 2. This invention integrates asynchronous event stream processing, dynamic feature weighted fusion, Kalman filter prediction, and Hungarian algorithm matching to enable real-time positioning, trajectory tracking, and motion status output for traffic participants such as vehicles, pedestrians, and cyclists. It can support obstacle avoidance, following, and lane changing functions of decision-making systems in unmanned systems, improving vehicle driving safety and environmental perception reliability. It is applicable to unmanned system vehicles in high-speed driving (speed > 60 km / h), complex lighting (strong light, backlight, tunnel entrances and exits), and target interaction (occlusion, merging, crossing), effectively improving the driving safety of unmanned system vehicles in complex environments.

[0091] 3. This invention can solve the technical problems of insufficient real-time performance and susceptibility to high-speed movement, target occlusion and environmental noise when existing algorithms track various traffic participants such as vehicles, pedestrians and cyclists in complex scenarios of unmanned systems, and can provide highly reliable obstacle dynamic perception support for unmanned system decision-making systems.

[0092] 4. This invention uses the asynchronous event stream output by the event camera as the core data input. First, it performs event data preprocessing, filtering isolated noise events caused by road surface reflection, sensor jitter, and raindrop interference. Then, it dynamically adapts a preset time window based on the target's movement speed, grouping the denoised event stream into event blocks. Next, it performs connected component analysis on the first valid event block to identify the target to be tracked, simultaneously extracting the target's static features (including bounding box parameters and contour moments) and dynamic features (including histogram of grayscale change direction and initial velocity vector). Finally, based on event density thresholds, motion direction angle change thresholds, and velocity... An event-driven triggering mechanism is constructed using a degree change rate threshold to enable rapid tracking and updates for potentially dangerous scenarios such as rapid acceleration, turning, and close-range movement. A correlation matrix is ​​then built using weighted similarity of static and dynamic features (dynamic features have a weight of 0.6, adapting to the motion-state-priority perception requirements of unmanned systems). The optimal match is then solved using the Hungarian algorithm, and Kalman filtering is introduced to predict the target's world coordinate system position, resolving ID conflicts in scenarios such as vehicle merging. Finally, the target's real-time position, trajectory, and tracking status (normal / occluded / lost) are updated, outputting the target ID, position, and speed information.

[0093] 5. This invention further optimizes the dynamic feature calculation method for the scene characteristics of unmanned systems, and uses optical flow vectors to replace traditional motion velocity vectors. This can accurately capture the high-speed motion state of vehicles, effectively adapt to complex road conditions such as high-speed driving, target occlusion, and vehicle merging, significantly improve the real-time performance and accuracy of multi-target tracking, and meet the high reliability requirements of unmanned systems for dynamic perception of obstacles. Attached Figure Description

[0094] Figure 1 This is a schematic diagram of the overall process of the present invention in Example 1;

[0095] Figure 2 This is a schematic diagram of the event data preprocessing process in Example 1;

[0096] Figure 3 This is a schematic diagram of the extraction and annotation of initial features for multiple targets in Example 1;

[0097] Figure 4 This is a schematic diagram illustrating the determination of the tracking triggering conditions in Example 1;

[0098] Figure 5 This is a flowchart illustrating the multi-target association and conflict resolution process in Example 1;

[0099] Figure 6 This is a comparative diagram illustrating the use of dynamic feature optimization in Example 1. Detailed Implementation

[0100] The present invention will be further described in detail below through specific embodiments, but this does not limit the scope of the present invention.

[0101] Example 1

[0102] An event-driven multi-target tracking algorithm for unmanned systems, such as... Figure 1 As shown, the specific implementation steps of USV-UAV cooperative control are as follows:

[0103] S1. Event Data Preprocessing: Obtain the asynchronous event stream output by the event camera, preprocess the asynchronous event stream, and generate event blocks.

[0104] In step S1, the asynchronous event stream is collected by the event camera mounted on the unmanned system vehicle, and each event is represented as:

[0105] (x,y,t,p);

[0106] Where x and y are pixel coordinates, t is a timestamp, and p∈{+1,-1} is the direction of grayscale change (reflecting the brightness changes between traffic participants and the background).

[0107] The asynchronous event stream is then denoised to filter out isolated noise events caused by road surface reflections, sensor jitter, or raindrop interference, as shown below:

[0108] For noise generated by strong light, shadow or road bumps in unmanned system scenarios, for an event (x,y,t,p), if the number of events in its 3×3 neighborhood that satisfy |t′-t|≤τ is less than N0, it is judged as a noise event and filtered out.

[0109] N0=ρ min ·9·τ;

[0110] Where N0 is the quantity threshold (3-5); ρ min τ represents the minimum event density for the scene, with a value of 0.5 events per (pixel·ms), corresponding to the event generation rate of slow-moving pedestrians; τ is the time threshold (0.1ms-0.5ms, adapted to the time resolution of high-speed moving targets).

[0111] The denoised event stream is grouped to generate event block E. k :

[0112] E k ={(x i ,y i ,t i ,p i |t∈[kΔt,(k+1)Δt)};

[0113] Where k is the event block index; Δt is the preset time window, which adapts to the movement speed of the target in the unmanned system scenario (e.g., a smaller Δt is needed for high-speed vehicles to avoid information loss).

[0114] The preset time window Δt satisfies Δt=max(0.5ms,1 / ρ avg );

[0115] Where, ρ avg The average event density in unmanned system scenarios is ρ, which is higher on highways due to the rapid movement of vehicles. avg Higher, Δt smaller.

[0116] The event block E k The minimum number of events is:

[0117]

[0118] Among them, S min Minimum target area, unit: pixel 2 Pixel size appropriate for children or small vehicles.

[0119] S2. Initial feature extraction for multiple targets: Extract the static and dynamic features of each target.

[0120] In step S2, for the first valid event block E0 in the event block, N targets of traffic participants to be tracked are identified through connected component analysis (N≥2, such as multiple cars and pedestrians at an intersection), and each target corresponds to a region. For each target n, extract static and dynamic features.

[0121] The static features include bounding box parameter B used to adapt to different contour sizes of vehicles and pedestrians. n =(x min ,x max ,y min ,y max ) and contour Hu moment h n =(h n1 ,h n2 ,...,h n7 ).

[0122] h n1 ,h n2 ,...,h n7 Represented as h ni h ni Let Hu be the i-th order moment, which is expressed as:

[0123] h ni =∫∫(x,y) i I(x,y)dxdy;

[0124] Where I(x,y) is the event density function of event block E0 at (x,y), that is, the number of events per unit area, used to distinguish between vehicles (high-density area) and pedestrians (medium-density area); x min The minimum value of the x-coordinate; x max The maximum value of the x-coordinate; y min The minimum value of the y-coordinate; y max y is the maximum value of the y-coordinate, corresponding to the upper and lower boundaries of the integration region in the x and y directions; dx and dy are the differential elements in the x and y directions, respectively, used to divide the two-dimensional region into infinitesimal elements;

[0125] The dynamic features include a histogram D representing the direction of grayscale changes that reflects the movement of the target surface texture (such as the difference between a vehicle body and a pedestrian's clothing). n and initial velocity

[0126] Represented as:

[0127]

[0128] in, For R n The center coordinates; For R n Average timestamp of internal events; |R n |for R n The number of events within the system, speed calculation adapted to the road coordinate system; v nx Corresponding to lateral motion, v ny Corresponding to longitudinal movement.

[0129] S3, Event-Driven Tracking Trigger: The obtained event density is used to trigger tracking updates for the target.

[0130] In step S3, for subsequent event block E k Real-time calculation of each target n in the current region Event density within;

[0131]

[0132] in, For target n in the kth event block region The event density; is R. n The area.

[0133]

[0134] in, Let be the change in the direction angle of motion of target n at time k and time k-1; This is the current direction angle, reflecting vehicle turning or pedestrian changing direction; Let be the direction angle of motion of target n at time k-1.

[0135]

[0136] in, Let n be the rate of change of relative velocity of target n at time k, reflecting rapid acceleration / deceleration; Let n be the velocity vector of target n at time k-1.

[0137] exist or or At that time, the tracking update of target n is triggered to quickly respond to potential dangers in unmanned systems.

[0138] Wherein, ρ0 is the first threshold, corresponding to the target entering the near field or undergoing violent movement; θ0 is the second threshold, corresponding to vehicles turning or pedestrians crossing the road; α0 is the third threshold, corresponding to emergency braking or acceleration.

[0139] The first threshold ρ0 = 50 pixels / (pixel·ms) corresponds to the target moving rapidly at close range, such as a vehicle approaching; the second threshold θ0 = 15° corresponds to the change in direction of a vehicle turning or a pedestrian crossing the road; the third threshold α0 = 25% corresponds to the rate of change of speed of a vehicle accelerating / decelerating rapidly.

[0140] S4. Multi-target association and conflict resolution: Extract real-time static and dynamic features of potential targets, determine the matching status, and use them for target tracking status updates and output.

[0141] In step S4, the event block E that triggered the update is... k Extract real-time static features of potential target m and real-time dynamic features Calculate the feature similarity S(n,m) with the historical target n:

[0142] S(n,m)=w s ·S s (n,m)+w d ·S d (n,m);

[0143] Among them, w s +w d =1, w s For static feature weights, w d For dynamic feature weights, the dynamic weights adapt to the priority of target motion in unmanned systems, w s =0.4,w d=0.6, with higher dynamic feature weights, playing a crucial role in adapting to target motion in unmanned systems; S s (n,m) represents the static feature similarity; S d (n,m) represents the dynamic feature similarity.

[0144] Static feature similarity is represented as:

[0145]

[0146] Where, ||·|2 is the Euclidean distance, used to match the contour consistency of vehicles / pedestrians; h n , These are the static feature vectors of targets n and m, respectively, with superscripts used to distinguish them from h. n Perform a similarity comparison.

[0147] Dynamic feature similarity is represented as:

[0148]

[0149] in,‖·‖ B D is the Barthelium distance, β is an adjustment parameter, β = 0.8, used to match motion patterns, such as the difference between a vehicle moving at a constant speed and a pedestrian moving at varying speeds; n , Let D represent the dynamic feature distributions of targets n and m, respectively. The superscript here is used to distinguish them from D. n Perform a similarity comparison;

[0150] An association matrix is ​​constructed based on feature similarity S(n,m), and the Hungarian algorithm is used to solve for the optimal match, assigning historical target IDs to potential targets. If there are unmatched targets with S(n,m) < γ0 (such as pedestrians suddenly appearing at intersections), γ0 is a fourth threshold, γ0 = 0.8, used to ensure the reliability of ID matching. The location of historical targets is predicted using Kalman filtering. (Adapted to road planar motion model), represented as:

[0151]

[0152] Where F is the state transition matrix and G is the input matrix.

[0153] Recalculate the Intersection over Union (IOU) ratio between the predicted location and the potential target to resolve ID conflicts (such as target confusion during vehicle merging).

[0154] Tracking status updates and output: Update the real-time positions of each target based on the correlation results. The road location mapped to the world coordinate system, and the trajectory used to predict the target driving path. Tracking status; output historical target IDs and historical target locations for all targets. speed Provide obstacle dynamic information for the decision-making system in unmanned systems.

[0155] The state transition matrix of the Kalman filter is:

[0156]

[0157] Its input matrix is:

[0158]

[0159] Its state vector is:

[0160] pos n =(p,q,v) x ,v y ) T ;

[0161] Where x, y represent the positions in the road coordinate system; v x ,v y It provides lateral and longitudinal velocities, adapting to the planar motion models of vehicles and pedestrians.

[0162] The tracking status includes normal tracking / occluded tracking / lost tracking;

[0163] The criteria for determining occlusion tracking are:

[0164] Intersection over Union (IOU) of the regions of target n and target m n ,R m When ) > 0.5 (e.g., a large vehicle obstructing a small vehicle), use the expression. Update the predicted location.

[0165] The criteria for determining "tracking loss" are:

[0166] If K consecutive event blocks fail to match (K=3), trigger a re-detection: for event block E k For events in the dataset that have not been assigned a historical target ID (which may be recurring targets), a candidate region C is generated, represented as:

[0167]

[0168] Where (p,q) are the spatial coordinates of the candidate region, (p',q') are the spatial coordinates of the event in the event block, t is the temporal attribute of the event, p is other attributes of the event, λ is the spatial decay factor, and η is the candidate region threshold. It adapts to the target distribution range of the road scene and matches the features of historically lost targets (such as vehicles that reappear after being occluded).

[0169] Dynamic feature optimization is performed on the high-speed motion characteristics of vehicle targets in unmanned systems: initial velocity... Replace with event-stream based optical flow vectors To more accurately reflect the relative motion of the road surface.

[0170] Optical flow vector Represented as:

[0171]

[0172] Where, x t+δ y t+δ Let x and y be the spatial coordinates of the event after a time interval δ, t be the temporal attribute of the event, p be other attributes of the event, and R be the spatial coordinates of the event. n The target area is defined by δ, which is the optical flow calculation time interval, set to 0.5ms, to adapt to instantaneous speed capture under high-speed vehicle movement.

[0173] Example 2

[0174] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step in an event-driven multi-target tracking algorithm for an unmanned system as described in Embodiment 1.

[0175] Furthermore, the event-driven multi-target tracking algorithm process for unmanned systems described in Embodiment 1 can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the functions defined in the method of this application.

[0176] Example 3

[0177] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any step in an event-driven multi-target tracking algorithm for an unmanned system as described in Embodiment 1.

[0178] The computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0179] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Python and C++, as well as conventional procedural programming languages ​​or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0180] In this embodiment, the computer-readable storage medium can be accelerated using hardware such as a GPU. The parallel computing advantage of the GPU is used to accelerate any step in the implementation of an event-driven multi-target tracking algorithm for unmanned systems as described in Embodiment 1.

[0181] In summary, this invention effectively overcomes the shortcomings of the prior art and has high industrial applicability. The above embodiments are intended to illustrate the substantive content of this invention, but are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the essence and scope of protection of this invention.

[0182] The above embodiments are specific implementations of the present invention, but the implementation of the present invention is not limited to the above embodiments. Any other combination, change, modification, substitution, or simplification that does not exceed the design concept of the present invention shall fall within the protection scope of the present invention.

Claims

1. An event-driven based multi-target tracking algorithm for unmanned systems, characterized in that, The method comprises the following steps: S1, event data preprocessing: obtaining an asynchronous event stream output by an event camera, preprocessing the asynchronous event stream, and generating an event block; S2, multi-target initial feature extraction: extracting static features and dynamic features of each target; S3, event-driven tracking triggering: obtaining event density for triggering tracking update of the target; S4, multi-target association and conflict resolution: extracting real-time static features and real-time dynamic features of a potential target, judging a matching state, and used for tracking state update and output of the target.

2. The event-driven based multi-target tracking algorithm for unmanned systems of claim 1, wherein, In step S1, the asynchronous event stream is collected by an event camera carried by an unmanned system vehicle, and each event is represented as: (x,y,t,p); Wherein, x and y are pixel coordinates, t is a timestamp, and p∈{+1,-1} is a gray scale change direction; The asynchronous event stream is further denoised, and is represented as: N0 = p min • 9 · τ; where N0 is a quantity threshold; p min is the minimum event density of the scene, corresponding to the event generation rate of low-speed pedestrians; τ is a time threshold; Grouping the denoised event stream, generating event blocks E k : E k = {(x i ,y i ,t i ,p i )|t∈[kΔt,(k+1)Δt)}; Wherein, k is an event block index; and Δt is a preset time window; The preset time window Δt satisfies Δt = max(0.5 ms, 1 / ρ avg ) wherein p avg is the average event density for the unmanned system scenario; The event block E k The lower limit of the number of events is: where S min is the minimum target area.

3. The event-driven based multi-target tracking algorithm for unmanned systems of claim 2, wherein, In step S2, for the first valid event block E0 in the event block, a target of N to-be-tracked traffic participants is identified through connected domain analysis, and each target corresponds to a region For each target n, static features and dynamic features are extracted; The static features include the bounding box parameter B. n =(x min ,x max ,y min ,y max ) and contour Hu moment h n =(h n1 ,h n2 ,...,h n7 ); h n1 ,h n2 ,...,h n7 denoted by h ni , h ni is the i-th order Hu moment, denoted by h ni = ∫∫(x, y) i I(x, y)dxdy; where I(x, y) is the event density function of the event block E0 at (x, y), i.e., the number of events in a unit area; x min is the minimum value of the x coordinate; x max is the maximum value of the x coordinate; y min is the minimum value of the y coordinate; y max is the maximum value of the y coordinate, corresponding to the upper and lower limit boundaries of the integral region in the x and y directions; dx and dy are the differential elements in the x and y directions, respectively, for dividing the two-dimensional region into infinitesimal elements. The dynamic features include a gray level change direction histogram D n and initial motion velocity is expressed as: wherein, is R n the center coordinates of the road segment; is R n the average timestamp of the events within the road segment; |R n |is R n the number of events within the road segment, the speed computation adapts the road coordinate system; v nx corresponds to lateral motion, v ny corresponds to longitudinal motion.

4. The event-driven based multi-target tracking algorithm for unmanned systems of claim 3, wherein, In step S3, the subsequent event block E k is calculated in real time for each target n in the current region . wherein, the event density for target n in the kth block of events region S n the area of R n the area of R wherein, is the change of the direction angle of motion of the target n at the kth moment and the (k-1)th moment; is the current direction angle of motion at the kth moment, reflecting the steering of the vehicle or the change of direction of the pedestrian; is the direction angle of motion of the target n at the (k-1)th moment; wherein, is the relative speed change rate of target n at the kth moment, reflecting the sudden acceleration / deceleration; is the speed vector of target n at the k-1th moment; In or or a tracking update of the target n is triggered for fast response to potential hazards in the unmanned system; Wherein, ρ0 is a first threshold value, corresponding to a target entering a near field or experiencing violent motion; θ0 is a second threshold value, corresponding to a vehicle turning or a pedestrian crossing the road; and α0 is a third threshold value, corresponding to emergency braking or acceleration.

5. The event-driven based multi-target tracking algorithm for unmanned systems of claim 4, wherein, In step S4, the event block E triggering the update is extracted k , real-time static features of the potential target m are extracted and real-time dynamic features The similarity S(n,m) with the features of the historical target n is calculated: S(n, m) = w s • S s (n, m) + w d • S d (n, m); wherein w s +w d = 1, w s is a static feature weight, w d is a dynamic feature weight, the dynamic weight adapting the priority of the target motion in the unmanned system; S s (n,m) is a static feature similarity; S d (n,m) is a dynamic feature similarity; The static feature similarity is represented as: where ||·||2 is the Euclidean distance; h n , are the static feature vectors of the target n, m, respectively, and the superscript is used for similarity comparison with h n . The dynamic feature similarity is represented as: where || · || is the Euclidean distance B is the Bhattacharyya distance, and β is a tuning parameter; D n , are the dynamic feature distributions of the target n, m, respectively, where the superscripts are used for similarity comparison with D n ​ Based on the feature similarity S(n,m), a correlation matrix is constructed, and the Hungarian algorithm is used to solve the optimal matching to assign the historical target ID to the potential target; if there is an unmatched target with S(n,m)<γ0, the historical target position is predicted through Kalman filtering is represented as: Wherein, F is a state transition matrix; and G is an input matrix; The intersection over union (IOU) of the predicted position and the potential target is recalculated to solve the ID conflict; Tracking state update and output: update real-time position of each target according to association result Map road position, trajectory in world coordinate system and tracking state; output history target ID, history target position of all targets speed Provide dynamic information of obstacles for decision system in unmanned system.

6. The event-driven based multi-target tracking algorithm for unmanned systems of claim 5, wherein, The state transition matrix of the Kalman filter is: The input matrix thereof is: The state vector thereof is: pos n = (p, q, v x , v y ) T ; where p, q are positions in the road coordinate system; v x ,v y are lateral and longitudinal velocities.

7. The event-driven based multi-target tracking algorithm for unmanned systems of claim 6, wherein, The tracking state includes normal tracking / occlusion tracking / lost tracking; The determination condition of the occlusion tracking is: Region Intersection over Union IOU(R n ,R m ) > 0.5, the expression Update the predicted position; The determination condition of the tracking loss is: Trigger re-detection when K consecutive blocks of events are unmatched (K=3): For events in block E k Generate candidate regions C for events that do not have a history target ID assigned, denoted as: Wherein, (p,q) is a spatial coordinate point in the candidate region, (p',q') is a spatial coordinate of an event in the event block, t is a time attribute of the event, p is another attribute of the event, λ is a spatial attenuation factor, η is a candidate region threshold value, and is adapted to the target distribution range of a road scene and matches the features of a historical lost target.

8. The event-driven based multi-target tracking algorithm for unmanned systems according to claims 1-7, characterized in that, Performing dynamic feature optimization, replacing initial motion velocity with event stream based optical flow vectors for more accurate reflection of relative motion of the road surface; Optical flow vector is represented as: where x is the initial spatial x coordinate of the event in the target region R n where x is the initial spatial x coordinate of the event in the target region R n where x is the initial spatial x coordinate of the event in the target region R t+δ where x is the initial spatial x coordinate of the event in the target region R t+δ where x is the initial spatial x coordinate of the event in the target region R n where x is the initial spatial x coordinate of the event in the target region R 9. An electronic device comprising a memory and a processor, said memory having stored thereon a computer program, characterized in that, The processor executes the computer program to implement any step in the event-driven unmanned system multi-target tracking algorithm according to any one of claims 1-8.

10. A computer-readable storage medium, characterized in that, The computer readable storage medium stores a computer program, and the computer program is executed by a computer processor to implement any step in the event-driven unmanned system multi-target tracking algorithm according to any one of claims 1-8.

Images