A multi-target tracking method and system across devices

Abstract

The application discloses a cross-device multi-target tracking method and system. The method comprises the following steps: converting vehicle driving data collected by a camera and a radar into trajectory data in a global coordinate system; dividing all perception devices into a plurality of adjacent device pairs; checking the upstream trajectory and the downstream trajectory formed by the adjacent device pairs in time dimension and space dimension in sequence; if the upstream trajectory and the downstream trajectory meet the check, splicing the trajectory sequence in time stamp order to fuse the global trajectory, otherwise, temporarily storing the trajectory as an isolated trajectory independent of the global trajectory. The system is used to implement the above method. The application has the advantages of low calculation complexity and real-time processing demand of large-scale perception network without feature extraction. The spatial similarity calculation methods are designed for the same type and different type devices, which covers the collaborative demand of mainstream perception devices in intelligent transportation scenarios and has strong compatibility.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent transportation, in particular to a cross-device multi-target tracking method and system. BACKGROUND

[0002] With the popularization of intelligent transportation systems, multi-camera collaborative tracking has become a core technical requirement in wide-area monitoring scenarios. In particular, in scenarios such as intelligent parks, transportation hubs, and urban roads, it is necessary to ensure continuous tracking and trajectory matching of vehicle targets across cameras.

[0003] However, the existing mainstream methods rely heavily on vehicle appearance ReID (Re-Identification) features for cross-camera target association, and have the following inherent defects:

[0004] (1) High computational complexity: Deep ReID feature extraction models have high computational complexity, which reduces system real-time performance and makes it difficult to meet the real-time processing requirements of large-scale monitoring networks.

[0005] (2) Poor environmental robustness: Appearance features are easily affected by changes in lighting, occlusion, weather conditions, and camera viewing angles, and have insufficient stability, resulting in decreased matching accuracy.

[0006] (3) Strong data dependency: A large amount of cross-camera target labeled data is required for model training, which has high labeling costs, and the model has limited generalization ability in new scenarios.

[0007] (4) Insufficient multi-device adaptation: There is no adaptation solution designed for the trajectory data differences between radar and camera and other heterogeneous perception devices, and there is no solution to the matching error problem caused by the mismatch in the number of trajectory points and the difference in sampling frequency in overlapping perception areas. SUMMARY

[0008] One of the purposes of the present application is to provide a cross-device multi-target tracking method that can solve at least one of the defects in the background art.

[0009] Another purpose of the present application is to provide a cross-device multi-target tracking system that can solve at least one of the defects in the background art.

[0010] To achieve the above at least one purpose, the technical solution adopted by the present application is as follows: a cross-device multi-target tracking method, comprising the following steps:

[0011] S100: converting vehicle driving data collected by the camera and the radar into trajectory data in a global coordinate system;

[0012] S200: dividing all perception devices into a plurality of adjacent device pairs based on the spatial coverage range and topological relationship of different perception devices;

[0013] S300: sequentially performing time dimension and space dimension check on the upstream trajectory and the downstream trajectory respectively formed by the adjacent device pair;

[0014] S400: if the upstream trajectory and the downstream trajectory in step S300 satisfy the time dimension and space dimension check, splicing the trajectory sequence in the time stamp order to fuse the global trajectory, otherwise, temporarily storing as an isolated trajectory independent of the global trajectory.

[0015] Preferably, the time dimension check on the upstream trajectory and the downstream trajectory in step S300 includes the following process:

[0016] S310: calculating the difference between the end time of the upstream trajectory and the start time of the downstream trajectory, and if the difference is within a preset connection threshold, performing the next step; wherein the upstream trajectory and the downstream trajectory allow negative overlap;

[0017] S320: based on the time range of the upstream trajectory and the downstream trajectory, calculating the time overlap value; if the upstream trajectory and the downstream trajectory have no perceptual overlap region, the time overlap value is assigned to 0; if the upstream trajectory and the downstream trajectory have a perceptual overlap region, performing minimum overlap duration judgment on the calculated time overlap value, and performing the next step on the upstream trajectory and the downstream trajectory satisfying the minimum overlap duration;

[0018] S330: according to the obtained time overlap value, calculating the proportion of the time overlap value in the duration of the upstream trajectory; if the calculated proportion is greater than a preset threshold, determining that the overlap region time matching of the upstream trajectory and the downstream trajectory is passed.

[0019] Preferably, the space dimension check on the adjacent device pair having a perceptual overlap region includes the following process:

[0020] S341: extracting trajectory points located in the overlap region from the upstream trajectory and the downstream trajectory respectively to form a candidate point pair set;

[0021] S342: performing distance threshold-based screening on the obtained candidate point pair set to obtain a longest common point pair sequence satisfying the distance threshold;

[0022] S343: calculating the spatial similarity of the overlap region scene according to the obtained longest common point pair sequence;

[0023] S344: if the calculated spatial similarity is less than a set overlap space threshold, determining that the spatial matching of the upstream trajectory and the downstream trajectory is passed.

[0024] Preferably, the spatial similarity S sim The calculation formula of the spatial similarity S is as follows:

[0025] ;

[0026] wherein, T U and T D represent the upstream trajectory and the downstream trajectory respectively, K represents the number of the longest common point pairs in the longest common point pair sequence, represents the point coordinate of the corresponding upstream trajectory in the mthlongest common point pair, represents the point coordinate of the corresponding downstream trajectory in the mthlongest common point pair.

[0027] Preferably, the spatial dimension verification for the adjacent device pair without the perceptual overlapping region comprises the following process:

[0028] S351: selecting the end point of the upstream trajectory and the start point of the downstream trajectory as the key matching point pair;

[0029] S352: calculating the spatial similarity of the non-overlapping region scene according to the obtained key matching point pair;

[0030] S353: if the calculated spatial similarity is less than the set splicing spatial threshold, determining that the spatial matching of the upstream trajectory and the downstream trajectory is passed.

[0031] Preferably, the spatial similarity S sim is calculated according to the following formula:

[0032] ;

[0033] wherein, (p U-end (X), p U-end (Y)) represents the end point coordinate of the upstream trajectory in the key matching point pair, (p D-start (X), p D-start (Y)) represents the start point coordinate of the downstream trajectory in the key matching point pair.

[0034] Preferably, for the adjacent device pair without the perceptual overlapping region, the spatio-temporal evolution consistency verification is performed by introducing the chain vector, and the specific process comprises the following steps:

[0035] S361: selecting the last two points p U(n-1) and p U(n) of the upstream trajectory, and the first two points p D1 and p D2 of the downstream trajectory;

[0036] S362: constructing a first spatio-temporal chain vector based on the spatio-temporal coordinates of the point p U(n-1) and the point p D2 ; constructing a second spatio-temporal chain vector based on the spatio-temporal coordinates of the point p U(n) and the point p D1 ;

[0037] S363: Calculate the cross product module length of the first time chain vector and the second space-time chain vector; if the absolute value of the calculated cross product module length is less than the set collineation threshold, it is determined that the space-time evolution consistency check of the upstream trajectory and the downstream trajectory is passed.

[0038] Preferably, in step S100, the vehicle driving data collected by the camera includes vehicle bounding box coordinates, detection confidence, vehicle type and timestamp; based on the homography matrix of the camera, the vehicle bounding box coordinates are converted from the pixel coordinate system to the global coordinate system to form trajectory data including vehicle bounding box global coordinates, timestamp, device ID, local trajectory ID and detection confidence; the vehicle driving data collected by the radar includes vehicle temporary ID, timestamp, spatial coordinates and speed; the obtained radar original trajectory data is optimized and converted to the global coordinate system to form trajectory data including global spatial coordinates, timestamp, device ID, local trajectory ID and speed.

[0039] Preferably, for the optimization of the radar original trajectory data, the following process is included: if there is a record interval of adjacent trajectory points in the original trajectory data of the same vehicle temporary ID exceeding the threshold, it is determined that the adjacent trajectory points correspond to different vehicles; then the original trajectory data is segmented at the position of the adjacent trajectory points, and the vehicle temporary ID is reassigned to the segmented trajectory segments; the number of trajectory points included in each original trajectory data collected by the radar is counted, and the trajectories with a length less than 5% of the quantile are removed, and the remaining trajectories are assigned with corresponding local trajectory IDs.

[0040] A cross-device multi-target tracking system for implementing the cross-device multi-target tracking method described above; including a local tracking module, a trajectory preprocessing module and a global tracker; the local tracking module is deployed on each camera for detecting vehicles within the field of view and converting to global coordinate system data; the trajectory preprocessing module is deployed on each radar for tracking the trajectories of vehicles within the field of view and converting to global coordinate system data; the global tracker obtains the data sent by the local tracking module and the trajectory preprocessing module, and is used to maintain the global trajectory information of all vehicles and perform corresponding global trajectory fusion.

[0041] Compared with the prior art, the application has the beneficial effects that:

[0042] (1) No feature extraction is required, the core calculation only involves coordinate conversion, space-time parameter statistics and vector operation, the calculation complexity is low, and the real-time processing demand of large-scale perception network can be met.

[0043] (2) The tracking process does not depend on the appearance features of the vehicle and is not affected by visual interference factors such as light, occlusion, and weather. At the same time, through time overlap quantification and spatial similarity analysis, the matching fluctuations caused by device sampling frequency differences and coordinate errors are effectively resisted, and the stability is excellent.

[0044] (3) The radar-camera coordinate unified system is innovatively constructed, the spatial similarity calculation method is designed for the same type and different type devices respectively, the collaborative demand of the mainstream perception devices in the intelligent transportation scene is covered, and the compatibility is strong. BRIEF DESCRIPTION OF DRAWINGS

[0045] Figure 1 The figure is a schematic diagram of the overall working steps of the present application.

[0046] Figure 2 The figure is a schematic diagram of the trajectories of the adjacent device pairs with the perception overlap region in the present application.

[0047] Figure 3 The figure is a schematic diagram of the trajectories of the adjacent device pairs without the perception overlap region in the present application. DETAILED DESCRIPTION

[0048] In the following, the present application will be further described in conjunction with specific embodiments. It should be noted that in the description of the present application, the description of the reference terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. In the present application, the illustrative description of the above terms should not be understood as necessarily referring to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described can be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art can combine and combine different embodiments or examples described in the present application.

[0049] In the description of the present application, it should be noted that for the orientation words such as the terms "center", "transverse", "longitudinal", "length", "width", "thickness", "up", "down", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inside", "outside", "clockwise", "counterclockwise", etc. The orientation and position relationship shown in the drawing is based on the orientation or position relationship shown in the drawing, and is only for the convenience of describing the present application and simplifying the description, and does not indicate or imply that the indicated device or element must have a particular orientation, be constructed and operated in a particular orientation, and cannot be understood as limiting the specific protection scope of the present application.

[0050] It should be noted that the terms "first", "second" and the like in the description and claims of this application are used for distinguishing between similar elements and not necessarily for describing a sequential or chronological order. The terms "first", "second", and the like can be understood as encompassing a "one", "two", or more of something and do not have to appear in any given order.

[0051] In the present application, unless specifically defined otherwise, the terms "mount", "connect", "connection", "fixed", and like terms are used broadly and encompass both direct and indirect connections, mechanical and electrical connections, connections through intermediate circuitry, and connections which are within or through intervening elements.

[0052] In the present application, unless specifically defined otherwise, the terms "on", "under", and the like in relation to the first and second features can include both direct contact and non-direct contact, where non-direct contact can include contact through additional features. Also, the terms "on", "above", and "top" in relation to the first and second features include both directly above and obliquely above, or simply that the first feature is higher than the second feature in terms of horizontal height. The terms "under", "below", and "bottom" in relation to the first and second features include both directly below and obliquely below, or simply that the first feature is lower than the second feature in terms of horizontal height.

[0053] The terms "comprise", "comprising", "include", "including", and the like in the specification and claims of this application are used in their open-ended sense, and can be used in their inclusive and non-exclusive sense, unless otherwise specifically noted.

[0054] One aspect of the present application provides a multi-target tracking method across devices, as shown in Figure 1 A preferred embodiment includes the following steps:

[0055] S100: Convert the vehicle driving data collected by the camera and the radar into trajectory data in the global coordinate system.

[0056] It can be understood that the camera and the radar are both common perception devices in a semi-closed scene, and due to the different types of the two perception devices, it is difficult to realize trajectory judgment and fusion by directly pairing the data collected by the two perception devices, so it is necessary to unify the data collected by the two perception devices. For the camera end, a pre-calibrated homography matrix can be loaded, which defines the projection relationship from the image pixel coordinate system to the global coordinate system, and is used for spatial positioning of the vehicle position; for the radar end, global conversion parameters are loaded to convert the radar local coordinates collected by the radar to the global coordinate system, so as to ensure the same coordinate system as the camera trajectory.

[0057] S200: Based on the spatial coverage range and the topological relationship of different perception devices, all perception devices are divided into a plurality of adjacent device pairs.

[0058] It can be understood that a plurality of perception devices are generally needed to be deployed in a semi-closed scene, in order to facilitate tracking of the vehicle trajectory, it is necessary to obtain the perception range of different perception devices, i.e. the spatial coverage range; based on the perception range of different perception devices, according to all driving paths that can be generated by the vehicle in the semi-closed scene, the topological relationship of different perception devices along the driving path is constructed. According to the obtained topological relationship of different perception devices, the perception devices can be divided into a plurality of adjacent device pairs.

[0059] S300: The upstream trajectory and the downstream trajectory formed by the adjacent device pairs are sequentially checked in time dimension and space dimension.

[0060] It can be understood that the data calculation amount of time dimension checking is smaller than that of space dimension checking, so the upstream trajectory and the downstream trajectory can be checked in time dimension first, which can preferentially exclude part of irrelevant trajectories and retain a small amount of upstream trajectories and downstream trajectories that may belong to the same vehicle driving; then the small amount of retained trajectory data is checked for spatial consistency based on the space dimension, so as to effectively reduce the calculation amount of checking and improve the checking efficiency.

[0061] S400: If the upstream trajectory and the downstream trajectory in step S300 meet the time dimension and space dimension checking, the trajectory sequence is spliced according to the time stamp order to fuse the global trajectory, otherwise it is temporarily stored as an isolated trajectory independent of the global trajectory.

[0062] It can be understood that, for the upstream track and the downstream track both carrying the timestamp and the device identifier, when the track fusion of the adjacent device pair is performed, two tracks meeting the time dimension and space dimension verification can be spliced according to the timestamp order to fuse into a global track; the device identifiers of the upstream track and the downstream track can be retained in the global track to distinguish different perception segments. The newly fused global track can continue to be matched with the track of the downstream adjacent device, and the above steps are repeated, and finally the whole-link track construction across multiple devices is realized. For the track for which no matching object is found, it can be temporarily stored independently of the global track, and if there is still no match within a preset time, it can be marked as completed tracking; for example, a vehicle drives a short distance within the camera perception range and then parks, which makes the driving track of the vehicle only perceived by one camera.

[0063] Compared with the vehicle track tracking method using ReID feature extraction in the prior art, the technical solution of the present application does not need to perform feature extraction, the core calculation only involves coordinate conversion, space-time parameter statistics and vector operation, the calculation complexity is low, and the real-time processing demand of a large-scale perception network can be met. The tracking process does not depend on the appearance feature of the vehicle and is not affected by visual interference factors such as light, shielding and weather; at the same time, through time overlap quantification and space similarity analysis, the matching fluctuation caused by the difference in device sampling frequency and coordinate error is effectively resisted, and the stability is excellent. At the same time, the technical solution of the present application innovatively constructs a radar-camera coordinate unified system, designs a space similarity calculation method for the same type and different type devices respectively, covers the cooperative demand of mainstream perception devices in the intelligent transportation scene, and has strong compatibility.

[0064] In the embodiment, when step S100 is performed, the camera uses a lightweight detector YOLOv11 or YOLO26 to perform real-time detection on the video frame, and then outputs the vehicle bounding box coordinates, the detection confidence and the vehicle type; then the camera can continuously track the target vehicle through the deployed local tracking module, generate a track segment with a local ID, and record the coordinates of the bottom center point of each frame of vehicle bounding box (representing the ground position of the vehicle) and the corresponding timestamp. Since the bottom center point coordinates of the vehicle bounding box output by the camera are pixel coordinates p=(u,v), the homography matrix H c The conversion of the pixel coordinates to the global coordinates is completed, and the global coordinate position P w =(X,Y) of the target vehicle on the ground plane is obtained, and the specific conversion formula is as follows:

[0065] .

[0066] wherein, represents the homogeneous coordinate form of the two-dimensional pixel coordinates (u,v) for matrix operation; X' = X / w, Y' = Y / w, w = 1

[0067] Based on the above transformation, the corresponding trajectory data of the camera in the global coordinate system can be obtained, including the global coordinate of the vehicle bounding box, the timestamp, the device ID, the local trajectory ID and the detection confidence.

[0068] In this embodiment, when step S100 is performed, the radar can receive real-time collection data of millimeter waves, including vehicle temporary ID, timestamp, spatial coordinates and speed, and then the collected data can be filtered to eliminate abnormal data caused by sensing errors, such as sudden change of speed and coordinates exceeding the road range. The filtered radar data can be optimized, and then the sensor local coordinate system of the radar is converted to the global coordinate system while the speed information is retained, and the corresponding trajectory data of the radar in the global coordinate system can be obtained, including the global spatial coordinates, the timestamp, the device ID, the local trajectory ID and the speed.

[0069] It can be understood that for the filtered original trajectory data collected by the radar, when tracking the same vehicle, the tracking trajectory may be split into multiple short trajectories due to signal fluctuations, which makes the radar output multiple short trajectories formed by the same vehicle as the trajectories of multiple vehicles when performing data output. At the same time, due to clutter and multipath collection, there may be very short false trajectories, i.e., the false trajectory only corresponds to a small number of trajectory points. This makes the calculation of false trajectories inefficient, and false trajectories may be misjudged as new trajectories triggering false association. Therefore, before performing global coordinate conversion of the radar, the original trajectory data collected by the radar needs to be optimized.

[0070] Specifically, the optimization of the original trajectory data of the radar includes the following processes:

[0071] Trajectory segmentation: if the record interval of adjacent trajectory points in the original trajectory data of the same vehicle temporary ID exceeds the threshold value, it is judged that the adjacent trajectory points correspond to different vehicles; then the original trajectory data is segmented at the position of the adjacent trajectory points, and the vehicle temporary ID is re-assigned to the segmented trajectory segment. The threshold value for judging the abnormality of adjacent trajectory points can be selected by the person skilled in the art according to actual needs, for example, the trajectory point collection interval time of the radar, such as 50ms.

[0072] Short trajectory elimination: count the number of trajectory points included in each original trajectory data collected by the radar, eliminate the trajectories whose length is less than 5% quantile, so as to exclude the invalid trajectories caused by perception noise; then assign the corresponding local trajectory ID to the remaining trajectories.

[0073] It can be understood that the optimization processing of the radar trajectory can quickly eliminate invalid data through simple interval threshold and length screening, and the processing efficiency can be further improved.

[0074] In the embodiment, when step S200 is performed, for the divided adjacent device pair, there are mainly two cases, one is that the two perception devices of the adjacent device pair are the same type of perception devices, such as two cameras or two radars; the other is that the two perception devices of the adjacent device pair are different types of perception devices, that is, the camera and the radar form the adjacent device pair. For the convenience of understanding, the subsequent content will be described taking the adjacent device pair formed by the different types of perception devices as an example.

[0075] It should be noted that based on the spatial coverage range and the topological relationship of different perception devices, there are not only adjacent device pairs, but also non-adjacent device pairs; when performing global trajectory matching, it is preferred to be performed between the trajectories of the adjacent device pairs, and the trajectories of the non-adjacent device pairs are only triggered to match when the strong space-time constraints are met. That is, through the strong space-time constraints, the trajectories of the non-adjacent device pairs are prevented from appearing in the case of long distance and no perception overlap area. Generally, the trajectory matching of the non-adjacent device pairs is not considered, so the subsequent content will be described taking the trajectory matching of the adjacent device pairs as an example.

[0076] In the embodiment, for the time dimension verification of the upstream trajectory T U and the downstream trajectory T D in step S300, consistency verification can be performed from two ways of time continuity and time overlap, and the specific process includes the following steps:

[0077] S310: calculate the difference Δt=t D-start -t U-end between the end time t U-end of the upstream trajectory T U and the start time t D-start of the downstream trajectory T D ; if the difference Δt is within the preset continuity threshold, the time overlap verification of the next step can be performed.

[0078] It can be understood that for the upstream trajectory T U and the downstream trajectory T D representing the continuous driving of the same vehicle, they can be continuous or intermittent, but the intermittent time is short; that is, the vehicle can be regarded as a trajectory that is basically unchangeable in the interval time. That is, the upstream trajectory T U and the downstream trajectory T D are the same vehicle trajectory, and the time interval between them is less than the preset time interval threshold.U and the downstream track T D Allowing negative overlap, then for the range of splicing threshold, allow negative value, the specific value can be selected by the actual needs of the skilled in the art, for example, can be taken [-2s, 5s]; then when -2s≤Δt≤5s, it means that the upstream track T U and the downstream track T D Satisfy the time splicing check, the next step of time overlap check can be performed.

[0079] S320: Based on the time range of the upstream track T U and the downstream track T D , calculate the time overlap value; if the upstream track T U and the downstream track T D have no perceptual overlap area, the time overlap value is assigned to 0; if the upstream track T U and the downstream track T D have perceptual overlap area, the minimum overlap duration judgment is performed on the calculated time overlap value, and the upstream track T U and the downstream track T D that meet the minimum overlap duration are executed to the next step.

[0080] It can be understood that the time range of the upstream track T U is [t U-start , t U-end ], and the time range of the downstream track T D is [t D-start , t D-end ]; then the calculation formula of the time overlap value T overlap is:

[0081] .

[0082] It should be noted that after the calculation of the time overlap value T overlap , there are two results; one is that the time overlap value T overlap > 0, and the other result is that the time overlap value T overlap = 0. For the case of time overlap value T overlap > 0, it can be judged that the upstream track T U and the downstream track T D have perceptual overlap area; for the case of time overlap value T overlap = 0, it can be judged that the upstream track T U and the downstream track T D have no perceptual overlap area.

[0083] For the upstream track T U and the downstream track T DIn the scenario of the existence of the overlap region of the perception, there may exist abnormal overlap situations, such as the situation of irrelevant vehicles accidentally passing the target vehicle, the situation of single-frame false overlap caused by shaking, and the like. Therefore, for the scenario of the existence of the overlap region of the perception, the minimum overlap duration judgment needs to be performed on the calculated time overlap T overlap Only when the calculated time overlap T overlap is not less than the minimum overlap duration, can the abnormal overlap situation be ruled out.

[0084] S330: According to the obtained time overlap value, the proportion of the time overlap value to the duration of the upstream trajectory T U is calculated; if the calculated proportion is greater than a preset threshold, it is determined that the overlap region time matching of the upstream trajectory T U and the downstream trajectory T D passes.

[0085] It can be understood that the time overlap index T index may be used to represent the ratio of the time overlap value T overlap to the duration of the upstream trajectory, and the calculation formula of the time overlap index T index may be represented as:

[0086] .

[0087] The specific value of the threshold for judging whether the overlap region time matching passes or not can be determined by the actual needs of those skilled in the art, for example, the value can be 10% to 30%.

[0088] It should be noted that the unit of the minimum overlap duration is s, and the time overlap index T index is dimensionless; the minimum overlap duration is used to judge whether the overlap region is an abnormal overlap, and the time overlap index T index ensures that there is enough overlap proportion between the upstream trajectory T U and the downstream trajectory T D for subsequent similarity calculation. In order to facilitate understanding, a specific example can be used for detailed description as follows.

[0089] Specifically, it is assumed that the duration of the upstream trajectory T U is 20s, the duration of the downstream trajectory T D is 15s, and there is an overlap region with a duration of 2s between them. It is assumed that the minimum overlap duration is 1s, and the threshold for judging whether the overlap region time matching passes or not is 30%. Based on the given conditions, it can be known that the upstream trajectory T U and the downstream trajectory T D meet the minimum overlap duration requirement; the time overlap index T index = 10%, and the overlap region time matching does not pass.

[0090] Assuming the duration of the upstream track T U is 5s, the duration of the downstream track T D is 6s, and there is an overlap region of 2s duration between them. Assuming the minimum overlap duration is 1s, and the threshold for judging whether the overlap region time matching passes is 30%. Then based on the given conditions, it can be known that the upstream track T U and the downstream track T D meet the minimum overlap duration requirement, the time overlap index T index =40%, and the overlap region time matching passes.

[0091] Assuming the duration of the upstream track T U is 1.5s, the duration of the downstream track T D is 1s, and there is an overlap region of 0.8s duration between them. Assuming the minimum overlap duration is 1s, and the threshold for judging whether the overlap region time matching passes is 30%. Then based on the given conditions, it can be known that the upstream track T U and the downstream track T D do not meet the minimum overlap duration requirement, but the time overlap index T index =53.3%.

[0092] It can be understood that, as can be known from the above examples, if only the minimum overlap duration is used to judge the overlap region time matching, then longer tracks may not have enough overlap regions for spatial similarity calculation, which significantly reduces the consistency judgment accuracy of the upstream track T U and the downstream track T D . If only the time overlap index T index is used to judge the overlap region time matching, then extremely short tracks may pass the matching due to jitter. Therefore, in this embodiment, the minimum overlap duration is first used to make an absolute judgment of the upstream track T U and the downstream track T D in the duration dimension, and then the time overlap index T index is used to make a relative judgment of the upstream track T U and the downstream track T D in the duration dimension, thereby ensuring the accuracy of subsequent spatial consistency judgment.

[0093] As can be known from the above content, the upstream track T U and the downstream track T D have two cases, one of which is that there is a perceived overlap region, and the other of which is that there is no perceived overlap region; for the convenience of understanding, the specific description of the spatial dimension verification will be made for the two cases respectively.

[0094] I. Spatial dimension verification of adjacent device pairs with perceived overlap region.

[0095] As shown in Figure 2 , the specific verification includes the following processes:

[0096] S341: Extracting the track points located in the overlapping area from the upstream track T U and the downstream track T D , respectively, to form a candidate point pair set.

[0097] As can be understood from Figure 2 and Figure 3 , since there is a sampling interval in the data collection of the target vehicle by the camera and the radar, the first track collected by the camera and the second track collected by the radar can be equivalent to continuous track points according to the sampling interval; then the perception overlapping area between the upstream track T U and the downstream track T D can be regarded as the overlapping area of multiple track points. In the perception overlapping area, the track points of the upstream track T U and the track points of the corresponding time dimension in the downstream track T D can be paired to form a candidate point pair, and then after all the track points in the perception overlapping area are paired, multiple candidate point pairs can be obtained, which can constitute a candidate point pair set.

[0098] It should be noted that since the sampling intervals of the camera and the radar can be different, for the perception device with a larger sampling interval, the track points in the perception overlapping area can be supplemented by interpolation to ensure that the number of track points of the camera and the radar in the perception overlapping area is basically consistent.

[0099] S342: Screening the obtained candidate point pair set based on a distance threshold to obtain a longest common point pair sequence satisfying the distance threshold.

[0100] As can be understood, the paired candidate point pairs are paired according to the time dimension, but they can not correspond in the spatial dimension, that is, there is a distance between the two track points (the spatial position of the target vehicle) in the paired candidate point pair; if the distance between the two track points in the paired candidate point pair exceeds the set distance threshold, it can be basically judged that the two track points do not correspond in space, and then the two track points can be removed. The specific value of the distance threshold can be selected by the person skilled in the art according to the actual needs, for example, the value of the distance threshold can be 2m, 3m, etc.

[0101] S343: Calculating the spatial similarity of the overlapping area scene according to the obtained longest common point pair sequence.

[0102] Specifically, the number of longest common point pairs in the longest common point pair sequence can be K, and the point coordinates of the corresponding upstream track T U in the mth longest common point pair are The downstream trajectory T corresponding to any m-th longest common point pair D The coordinates of the point are Then the upstream trajectory T U and downstream trajectory T D Spatial similarity S between perceived overlapping regions sim The calculation formula is as follows:

[0103] .

[0104] S344: If the calculated spatial similarity S sim If the overlap space threshold is less than the set threshold, the upstream trajectory T is determined. U and downstream trajectory T D Spatial matching passed.

[0105] It is understandable that, based on the spatial similarity S mentioned above... sim The calculation formula shows that spatial similarity S sim It can be viewed as the average of the Euclidean distances of all longest common pairs of points. Therefore, based on the definition of Euclidean distance, the spatial similarity S... sim The larger the value, the greater the distance between the two points, indicating poorer spatial consistency. Therefore, in this embodiment, an overlap spatial threshold can be set to improve the spatial similarity S. sim The upstream trajectory T can only be determined if it is less than the overlap space threshold. U and downstream trajectory T D Spatial matching was successful. The specific value of the overlap spatial threshold can be selected according to the actual needs of those skilled in the art; for example, it can be 3m.

[0106] II. Spatial dimension verification for adjacent device pairs in areas of non-perceptual overlap.

[0107] like Figure 3 As shown, the specific verification process includes the following steps:

[0108] S351: Select the upstream trajectory T U The end point and the downstream trajectory T D The starting point is used as the key matching point pair.

[0109] It is understandable that the upstream trajectory T U The endpoint is the location where the target vehicle leaves the sensing range of the upstream sensing device; the downstream trajectory T D The starting point is the location point when the target vehicle just enters the sensing range of the downstream sensing device.

[0110] S352: Calculate the spatial similarity of non-overlapping scene pairs based on the obtained key matching point pairs.

[0111] Specifically, the end point coordinate of the upstream trajectory T U is (p U-end (X), p U-end (Y)), and the start point coordinate of the downstream trajectory T D is (p D-start (X), p D-start (Y)). The calculation formula of the spatial similarity S U of the non-aware overlapping region of the upstream trajectory T D and the downstream trajectory T sim is as follows:

[0112] .

[0113] S353: If the calculated spatial similarity S sim is less than the set splicing spatial threshold, it is determined that the spatial matching of the upstream trajectory T U and the downstream trajectory T D passes.

[0114] It can be understood that, based on the calculation formula of the spatial similarity S sim , the spatial similarity S sim can be regarded as the Euclidean distance of the key matching point pair; then based on the definition of the Euclidean distance, the greater the spatial similarity S sim , the farther the interval between the two points, that is, the poorer the spatial consistency, so in this embodiment, the splicing spatial threshold can be set so that the spatial matching of the upstream trajectory T sim and the downstream trajectory T U passes only when the spatial similarity S D is less than the splicing spatial threshold. The specific value of the splicing spatial threshold can be selected by the person skilled in the art according to the actual needs, for example, it can be 10m.

[0115] It should be noted that, based on the spatial dimension verification of the non-aware overlapping region of the upstream trajectory T U and the downstream trajectory T D , only one single point is used to determine the dimension between the upstream trajectory T U and the downstream trajectory T D , which results in a slightly poor spatial dimension verification accuracy of the upstream trajectory T U and the downstream trajectory T D in the non-aware overlapping region scenario. Therefore, in order to improve the spatial dimension verification accuracy of the upstream trajectory T U and the downstream trajectory T D in the non-aware overlapping region scenario, a chain vector can be introduced for spatiotemporal evolution consistency verification.

[0116] In this embodiment, the specific process of performing spatiotemporal evolution consistency verification on adjacent device pairs in non-overlapping regions using chain vectors is as follows:

[0117] S361: Select the upstream trajectory T U The last two points p U(n-1) and p U(n) and downstream trajectory T D The two starting points p D1 and p D2 .

[0118] S362: Based on point p U(n-1) and point p D2 Based on the spatiotemporal coordinates of point p, construct the first spatiotemporal chain vector; U(n) and point p D1 The spatiotemporal coordinates are used to construct a second spatiotemporal chain vector.

[0119] It is understandable that the spatiotemporal coordinates of a trajectory point consist of two-dimensional spatial coordinates and one-dimensional time coordinates; where point p U(n-1) The spacetime coordinates are (p U(n-1) (X), p U(n-1) (Y), t U(n-1) ), point p U(n) The spacetime coordinates are (p U(n) (X), p U(n) (Y), t U(n) ), point p D1 The spacetime coordinates are (p D1 (X), p D1 (Y), t D1 ), point p D2 The spacetime coordinates are (p D2 (X), p D2 (Y), t D2 Based on the coordinates described above, the expressions for the first spatiotemporal chain vector L1 and the second spatiotemporal chain vector L2 are as follows:

[0120] .

[0121] .

[0122] S363: Calculate the cross-product magnitude of the first time chain vector and the second spatiotemporal chain vector; if the absolute value of the calculated cross-product magnitude is less than the set collinearity threshold, determine the upstream trajectory T. U and downstream trajectory T D The spatiotemporal evolution consistency check passed.

[0123] It can be understood that the specific value of the collinear threshold can be selected by the person skilled in the art according to actual needs. Generally speaking, the closer the cross product module length is to 0, the higher the possibility that the upstream trajectory T U and the downstream trajectory T D are closer to collinearity in space-time evolution, i.e., belong to the same vehicle; therefore, the specific value of the collinear threshold can be 0.1-0.2.

[0124] Specifically, the calculation formula of the cross product module length |P| of the first space-time chain vector L1 and the second space-time chain vector L2 is as follows:

[0125] .

[0126] .

[0127] .

[0128] .

[0129] wherein P X , P Y and P Z respectively represent the three components of the space-time cross product in the spatial coordinates, and are respectively used to evaluate the non-collinear degree of the first space-time chain vector L1 and the second space-time chain vector L2 in three space-time directions.

[0130] It should be known that, in the technical solution of the present application, for the upstream trajectory T U and the downstream trajectory T D with the perception overlap region, the time overlap index and the spatial similarity are used to realize quantitative matching; for the upstream trajectory T U and the downstream trajectory T D without the perception overlap region, the chain vector cross product module length is used to check the space-time evolution consistency, forming a full-scene and multi-dimensional matching logic, which greatly improves the trajectory correlation accuracy.

[0131] Another convenient embodiment of the present application provides a cross-device multi-target tracking system for executing the cross-device multi-target tracking method described above; wherein one preferred embodiment includes a local tracking module, a trajectory preprocessing module, and a global tracker. The local tracking module is deployed at each camera and is used to detect vehicles within the field of view and convert them into global coordinate system data; the trajectory preprocessing module is deployed at each radar and is used to track the trajectories of vehicles within the field of view and convert them into global coordinate system data; the global tracker obtains the data sent by the local tracking module and the trajectory preprocessing module, and is used to maintain the global trajectory information of all vehicles and execute corresponding global trajectory fusion.

[0132] It can be understood that at the system startup, all camera and radar basic parameters are read, including device frame rate, time synchronization offset, camera homography matrix, radar coordinate conversion coefficient, etc. Then the global coordinate conversion is performed through the local tracking module deployed in the camera and the trajectory preprocessing module deployed in the radar; the converted global coordinate, timestamp, device ID, local trajectory ID, detection confidence (camera), speed (radar), etc. Information is packaged as structured observation data as the input source of the global tracker. Then the global tracker executes the above multi-target tracking method according to the input source, and finally outputs the complete trajectory of the target vehicle in the global coordinate system.

[0133] The above describes the basic principles, main features and advantages of the present application. Those skilled in the art should understand that the present application is not limited to the above embodiments, and the above embodiments and descriptions in the specification are only the principles of the present application. Without departing from the spirit and scope of the present application, various changes and improvements can be made to the present application, and these changes and improvements all fall within the scope of the claimed present application. The scope of protection claimed by the present application is defined by the appended claims and their equivalents.

Claims

1. A method for multi-target tracking across devices, the method comprising: The method comprises the following steps: S100: converting the vehicle driving data collected by the camera and the radar into trajectory data in a global coordinate system respectively; S200: dividing all the perception devices into a plurality of adjacent device pairs based on the spatial coverage range and the topological relationship of different perception devices; S300: sequentially checking the time dimension and the spatial dimension of the upstream trajectory and the downstream trajectory formed by the adjacent device pairs respectively; S400: if the upstream trajectory and the downstream trajectory in step S300 satisfy the time dimension and the spatial dimension check, splicing the trajectory point sequence in the time stamp order to fuse the global trajectory, otherwise, temporarily storing as an isolated trajectory independent of the global trajectory; In step S300, for the adjacent device pairs without a perception overlap region, time-space evolution consistency check is performed by introducing a chain vector, which specifically includes the following processes: S361: Select the last two points p of the upstream trajectory U(n-1) and p U(n) and the two starting points p of the downstream trajectory D1 and p D2 ; S362: constructing a first spatiotemporal chain vector based on the spatiotemporal coordinates of point p U(n-1) and point p D2 S364: constructing a second spatiotemporal chain vector based on the spatiotemporal coordinates of point p U(n) and point p D1 ​ S363: calculating the cross product module length of the first time chain vector and the second time-space chain vector; if the absolute value of the calculated cross product module length is less than a set collinear threshold, it is determined that the time-space evolution consistency check of the upstream trajectory and the downstream trajectory is passed.

2. The cross-device multi-target tracking method of claim 1, wherein, The time dimension check of the upstream trajectory and the downstream trajectory in step S300 includes the following processes: S310: calculating the difference between the end time of the upstream trajectory and the start time of the downstream trajectory, if the difference is within a preset connection threshold, the next step is performed; wherein the upstream trajectory and the downstream trajectory allow negative overlap; S320: based on the time range of the upstream trajectory and the downstream trajectory, calculating the time overlap value; if the upstream trajectory and the downstream trajectory have no perception overlap region, the time overlap value is assigned to 0; if the upstream trajectory and the downstream trajectory have a perception overlap region, the minimum overlap duration is judged for the calculated time overlap value, and the next step is performed for the upstream trajectory and the downstream trajectory that satisfy the minimum overlap duration; S330: according to the obtained time overlap value, calculating the proportion of the time overlap value in the upstream trajectory duration; if the calculated proportion is greater than a preset threshold, it is determined that the overlap region time matching of the upstream trajectory and the downstream trajectory is passed.

3. The cross-device multi-target tracking method of claim 2, wherein, The spatial dimension check for the adjacent device pairs with a perception overlap region includes the following processes: S341: extracting the trajectory points located in the overlap region from the upstream trajectory and the downstream trajectory respectively to form a candidate point pair set; S342: screening the obtained candidate point pair set based on a distance threshold to obtain the longest common point pair sequence that satisfies the distance threshold; S343: calculating the spatial similarity of the overlap region scene according to the obtained longest common point pair sequence; S344: if the calculated spatial similarity is less than a set overlap space threshold, it is determined that the spatial matching of the upstream trajectory and the downstream trajectory is passed.

4. The cross-device multi-target tracking method of claim 3, wherein, The spatial similarity S in step S343 sim The calculation formula is as follows: ； wherein T U and T D respectively represent the upstream trajectory and the downstream trajectory, K represents the number of the longest common point pairs in the longest common point pair sequence, represents the point coordinate corresponding to the upstream trajectory in the mth longest common point pair, represents the point coordinate corresponding to the downstream trajectory in the mth longest common point pair.

5. The cross-device multi-target tracking method of claim 2, wherein, The spatial dimension check for the adjacent device pairs without a perception overlap region includes the following processes: S351: selecting the end point of the upstream trajectory and the start point of the downstream trajectory as the key matching point pair; S352: calculating the spatial similarity of the non-overlap region scene according to the obtained key matching point pair; S353: if the calculated spatial similarity is less than a set connection space threshold, it is determined that the spatial matching of the upstream trajectory and the downstream trajectory is passed.

6. The cross-device multi-target tracking method of claim 5, wherein, The spatial similarity S in step S352 sim The calculation formula is as follows: ； where T U and T D represent the upstream and downstream trajectories, respectively, (p U-end (X), p U-end (Y)) represent the end point coordinates of the upstream trajectory in the key matching point pair, and (p D-start (X), p D-start (Y)) represent the start point coordinates of the downstream trajectory in the key matching point pair.

7. The cross-device multi-target tracking method of any one of claims 1-6, wherein, In step S100, the vehicle driving data collected by the camera includes vehicle bounding box coordinates, detection confidence, vehicle type, and timestamp; Based on the camera's homography matrix, the vehicle bounding box coordinates are converted from the pixel coordinate system to the global coordinate system, forming trajectory data including vehicle bounding box global coordinates, timestamp, device ID, local trajectory ID, and detection confidence; The vehicle driving data collected by the radar includes vehicle temporary ID, timestamp, spatial coordinates, and speed; After optimizing the obtained radar raw trajectory data, it is converted to the global coordinate system, forming trajectory data including global spatial coordinates, timestamp, device ID, local trajectory ID, and speed.

8. The cross-device multi-target tracking method of claim 7, wherein, For the optimization of radar raw trajectory data, the following processes are included: If the record interval of adjacent trajectory points in the raw trajectory data of the same vehicle temporary ID exceeds the threshold, it is determined that the adjacent trajectory points correspond to different vehicles; then the raw trajectory data is segmented at the position of the adjacent trajectory point, and the vehicle temporary ID is reassigned to the segmented trajectory segment; The number of trajectory points included in each raw trajectory data collected by the radar is counted, and trajectories with a length less than 5% of the quantile are removed. The remaining trajectories are assigned corresponding local trajectory IDs.

9. A cross-device multi-target tracking system for implementing the cross-device multi-target tracking method of any one of claims 1-8; characterized by, It includes: a local tracking module; The local tracking module is deployed on each camera and is used to detect vehicles within the field of view and convert them to global coordinate system data; a trajectory preprocessing module; the trajectory preprocessing module is deployed on each radar and is used to track vehicles within the field of view and convert them to global coordinate system data; and a global tracker; The global tracker obtains data sent by the local tracking module and the trajectory preprocessing module, and is used to maintain global trajectory information of all vehicles and perform corresponding global trajectory fusion.

Images