Intersection cross-domain multi-target tracking system and method based on C-V2X and thundersight fusion

Through the fusion technology of C-V2X and radar, combined with YOLOv10 network and multi-model extended Kalman filtering method, the perceived blind spot and precise position acquisition problems of multi-objective tracking in intelligent transportation systems are solved, and seamless tracking of cross-domain multi-objectives is achieved and efficient identification improve the system's environmental perception and reliability.

CN118967748AActive Publication Date: 2024-11-15CHONGQING UNIV OF POSTS & TELECOMM

Patent Information

Application Number
CN202411038914.7
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-31
Publication Date
2024-11-15
Estimated Expiration
2044-07-31

AI Technical Summary

Technical Problem

The multi-objective tracking technology in the existing intelligent transportation system has the problem of perceived blind spots and the shortcomings of traditional visual object detection methods in obtaining the precise location of objects, making it difficult to achieve seamless tracking of cross-domain multi-objectives.

Method used

Using the fusion technology based on C-V2X and radar, radar and through the target detection module, C-V2X perception compensation module, target appearance feature re-identification module and trajectory estimation module, combined with the YOLOv10 network and multi-model extended Kalman filtering method, radar and camera are realized information fusion and cross-domain multi-objective tracking.

Benefits of technology

Improve the environmental perception capability of intelligent transportation systems and the reliability of cross-domain multi-objective tracking, enhance the re-identification ability of vehicles of different sizes and types, overcome sensor failure problems caused by weather and environmental factors, and make up for the vision of a single sensor. The blind spot problem improves the overall performance of the system.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN118967748A_ABST
    Figure CN118967748A_ABST
Patent Text Reader

Abstract

The invention relates to an intersection cross-domain multi-target tracking system and method based on C-V2X and thunder-vision fusion, and belongs to the field of vehicle-road cooperation and intelligent traffic. The system comprises a target detection module, a C-V2X perception compensation module, a target appearance feature re-identification module, a non-perception area trajectory estimation module and a cross-domain multi-target tracking module, fusion perception is performed on a vehicle by using a roadside camera and a radar, and a cascade matching strategy of a target appearance feature re-identification model and multi-source information fusion is provided. The adaptive vehicle trajectory prediction method based on multi-model extended Kalman filtering is provided for solving the problem of vehicle trajectory interruption caused by the fact that visual information of a vehicle cannot be extracted in a view blind area, and the attitude of the vehicle in the blind area is estimated. According to the invention, the environment perception capability of the intelligent traffic system and the reliability of cross-domain multi-target tracking are improved, and technical guarantee is provided for safer and more efficient intelligent traffic.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the field of vehicle-road collaboration and intelligent transportation, and relates to a cross-domain multi-target tracking system and method at an intersection based on C-V2X and radar-vision fusion. Background Art

[0002] With the development of C-V2X technology, vehicle-road cooperative systems have made significant progress in recent years. The management of traffic intersections has also gradually developed towards the direction of intelligent intersections with multi-dimensional coordination of "people-vehicle-road-cloud", becoming an important part of the future intelligent transportation system. In these systems, environmental perception is the core foundation, and the accuracy and continuity of its target perception directly affect the performance and function of subsequent subsystems. With the development of computer vision and deep learning technology, environmental perception algorithms based on large amounts of data input such as video streams have gradually been applied to actual scenarios. Among them, multi-target tracking algorithms have become an important tool in the field of intelligent transportation after target detection algorithms. At present, the application of multi-target tracking technology in intelligent transportation systems still faces many challenges. Traditional visual target detection methods are mostly based on deep learning. Although they can detect traffic objects, they still have shortcomings in obtaining the precise location of objects.

[0003] The multi-target tracking method integrating C-V2X communication can not only expand the perception range, but also provide accurate location information, thus providing more powerful support for traffic management, vehicle decision-making and driving safety. The present invention proposes a cross-domain multi-target tracking system and method at an intersection based on C-V2X and radar vision fusion, aiming to solve the perception blind spot problem existing in the prior art and realize seamless tracking of cross-domain multi-targets. Summary of the invention

[0004] In view of this, the purpose of the present invention is to provide a cross-domain multi-target tracking system and method at an intersection based on C-V2X and radar-vision fusion, so as to improve the environmental perception capability of the intelligent transportation system and the reliability of cross-domain multi-target tracking.

[0005] To achieve the above objectives, the present invention provides, on the one hand, a cross-domain multi-target tracking system for intersections based on C-V2X and radar vision fusion, the system comprising a target detection module, a C-V2X perception compensation module, a target appearance feature re-identification module, a trajectory estimation module and a cross-domain multi-target tracking module.

[0006] The target detection module obtains the appearance information, motion information and spatial information of the vehicle on the road based on the camera and radar, and performs coordinate conversion between world coordinates, camera coordinates, image coordinates and pixel coordinates to achieve the fusion of radar targets and visual targets. Among them, the spatial information includes the longitude and latitude of the target, and the motion information includes the speed, acceleration, angular velocity, yaw angle, etc. of the target.

[0007] The C-V2X perception compensation module is used to perform perception compensation on the target detection module to increase the accuracy of fusion target detection;

[0008] The target appearance feature re-identification module is used to extract appearance features from the appearance information obtained by the target detection module, and identify the target vehicle in the video sequences of multiple non-overlapping field of view cameras;

[0009] The trajectory estimation module is used to estimate the trajectory and posture of the vehicle within the blind area of ​​the camera's field of view to ensure the consistency of the global ID of the tracked target;

[0010] The cross-domain multi-target tracking module completes the multi-target tracking task under the single camera field of view according to the output of the target detection module and the C-V2X perception compensation module, and then combines the target appearance feature re-identification module and the trajectory estimation module to complete the data association and identity matching of the target trajectory under different fields of view, thereby realizing cross-domain multi-target tracking.

[0011] Furthermore, the target detection module acquires the target by integrating the camera and the radar based on the YOLOv10 network.

[0012] Furthermore, the C-V2X perception compensation module integrates V2X communication to perform perception compensation based on the target detection module to make up for the target loss problem caused by the failure of the roadside sensor; the C-V2X perception compensation module performs perception compensation according to different situations of whether there is a camera target and whether there is a radar target.

[0013] Furthermore, the target appearance feature re-identification module includes a global feature branch, a local feature branch and a multi-scale feature branch. The feature vectors of the global feature map are extracted through the three feature extraction branches, and the feature vectors are then concatenated to obtain a feature representation.

[0014] Furthermore, the cross-domain multi-target tracking module is based on a cascade matching strategy of multi-source information fusion, fully utilizes the target appearance features, spatial features and motion features, and uses an elliptical adaptive association gate to increase the accuracy of data association, and dynamically adjusts the size and shape of the association gate according to the speed and uncertainty of the fused target to cover more potential positions of the target.

[0015] On the other hand, the present invention provides a cross-domain multi-target tracking method at an intersection based on C-V2X and radar-visual fusion, which obtains radar targets and visual targets through radar and camera respectively, and fuses radar targets and visual targets to obtain appearance information, motion information and spatial information of vehicles on the road;

[0016] C-V2X communication data is used to compensate for the perception of the fused detection targets of radar and camera to make up for the target detection failure caused by the failure of roadside sensors;

[0017] Re-identify the appearance features from the appearance information of the vehicle, so as to identify the target vehicle in the video sequences of multiple non-overlapping field of view cameras;

[0018] In the field of view of a single camera, local trajectories of vehicles in the field of view are predicted; for vehicles in non-camera blind spots, the local trajectories of the vehicles are predicted by a single-camera multi-target tracking algorithm; for vehicles in the blind spots of the camera, the local trajectories of the vehicles are predicted by an adaptive vehicle trajectory prediction method based on multi-model extended Kalman filtering, ensuring the consistency of the global ID of the tracked target;

[0019] In the field of view of multiple different cameras, a cascade matching strategy based on multi-source information fusion is used to complete the data association and identity matching of target trajectories in different fields of view, thereby realizing cross-domain multi-target tracking. Among them, the appearance features, spatial features, and motion features of the vehicle are utilized, and an elliptical adaptive association gate is used to increase the accuracy of data association. At the same time, the size and shape of the association gate are dynamically adjusted according to the speed and uncertainty of the fused target to cover more potential positions of the target.

[0020] Furthermore, the perception compensation of the radar and camera fusion detection targets through C-V2X communication data includes the following:

[0021] When there are camera targets and radar targets, the camera targets and radar targets are matched and fused at the decision layer, and the BSM message is used to further verify whether the fused target is reasonable;

[0022] When there is a camera target but no radar target, a radar target is constructed using the vehicle position and speed parsed from the BSM message and fused with the camera target;

[0023] When there is no camera target but there is a radar target, the vehicle position and speed parsed from the BSM message are matched with the radar target, and then an adaptively sized target detection box is constructed in the pixel image according to the target size, type, and distance between the target and the camera to compensate for the camera target;

[0024] When there is no camera target and no radar target, V2X data is relied upon for perception compensation, with BSM messages acting as radar targets, and the camera target is compensated using the compensation method used when there is no camera target but there is a radar target.

[0025] Furthermore, extracting appearance features from the appearance information of the vehicle includes: extracting feature vectors of the global feature map through a global feature branch, a local feature branch, and a multi-scale feature branch, and then concatenating the feature vectors to obtain a feature representation, wherein:

[0026] The global feature branch assigns different attention weights to different positions of the global feature map through a spatial attention layer, and the attention weight of each position depends on the total activation degree of all channels at that position; then the global feature map is optimized through triplet loss and ID loss;

[0027] The local feature branch divides the global feature map into multiple parts, and each part uses a channel attention layer to emphasize the most recognizable channel in the feature map, thereby reducing noise interference; at the same time, the local features are weighted to increase attention to significant vehicle information and reduce noise interference;

[0028] The multi-scale feature branch divides the global feature map into four parts, and constructs a pyramid from coarse to fine to learn local features of different scales.

[0029] Furthermore, the adaptive vehicle trajectory prediction method based on multi-model extended Kalman filtering includes:

[0030] 1) Four extended Kalman filter models are used, including a constant position model, a constant speed model, a constant acceleration model, and a constant turning model; the constant position model is used to describe the state of a vehicle waiting for a traffic light, the constant speed model and the constant acceleration model are used to describe the state of a vehicle traveling in a straight line, and the constant turning model is used to describe the state of a vehicle turning;

[0031] 2) Parse the BSM message in each time step to obtain the vehicle's motion state information, and select X as the vehicle's state vector:

[0032]

[0033] In the formula, x and y represent the position coordinates of the vehicle, v represents the speed, a represents the acceleration, θ represents the yaw angle, and ω represents the angular velocity;

[0034] 3) Based on the characteristics of different motion models, the state transfer matrix is ​​constructed and the parameters of the extended Kalman filter model are initialized;

[0035] 4) Predefine a probability transfer matrix Π = [π ij ], combining the observed data of the current time step to dynamically select the most appropriate model for the next prediction; π ij represents the probability of switching from model i to j;

[0036] 5) Estimate the trajectory and motion state of the vehicle based on the predicted model;

[0037] 6) Collect the model state transfer data at different time steps during system operation, record the model actually used in each time step and the model of the next time step, and dynamically update the probability transfer matrix based on the observation data of the historical time step.

[0038] Furthermore, the cascade matching strategy includes:

[0039] 1) Use the appearance feature distance of the target to build a cost matrix for primary matching. Targets with appearance feature distances less than the threshold are judged as different targets, and an elliptical adaptive association gate is used to further screen targets that can be successfully matched;

[0040] 2) Define the association gate as:

[0041]

[0042] In the formula, (u 0 ,v 0 ) represents the center of the ellipse, i.e., the pixel coordinates of the target, a and b represent the radii of the major axis and minor axis respectively; the dynamic adjustment strategy of the association gate is:

[0043] Assume the target's velocity is v, split it into v x and v y , the greater the target speed, the longer the major axis of the associated gate should be to accommodate the target's potential larger movement range. The relationship between the adjustment of a and speed is:

[0044]

[0045] In the formula, a 0 represents the radius of the major axis of the base, k v represents the adjustment factor;

[0046] Using the covariance matrix Σ as the uncertainty matrix, the greater the uncertainty of the target, the wider the minor axis of the associated gate should be to cover more potential positions. The relationship between the adjustment of b and the uncertainty is:

[0047]

[0048] Where b 0 represents the basic minor axis radius, k ∑ represents the adjustment factor, λ max (Σ) represents the maximum eigenvalue of the covariance matrix, indicating the direction of maximum uncertainty;

[0049] 3) Construct an elliptical adaptive correlation gate centered on all detections;

[0050] 4) Assume that the target is from the camera field of view V 1 Move through the blind area to the camera field of view V 2 , project all predicted trajectories in the blind area to V 2 , perform secondary matching on the trajectories falling within the associated gate;

[0051] 5) Use the spatial features and motion features of the target for secondary matching and perform linear weighting:

[0052] The spatial position distance between the i-th detection and the j-th trajectory is used as the spatial feature similarity, the speed difference between the detection and the trajectory is used as the motion feature similarity, and the spatial feature similarity and the motion feature similarity are linearly weighted to construct a cost matrix for secondary matching.

[0053] The beneficial effects of the present invention are as follows: the present invention makes full use of intelligent roadside sensors and C-V2X communication technology, enhances the ability to re-identify vehicles of different sizes and types, improves the recognition of re-identification features, overcomes the problem of sensor failure caused by weather and environmental factors, and compensates for the blind spot problem of a single sensor. The overall system improves the environmental perception capability of the intelligent transportation system and the reliability of cross-domain multi-target tracking through multi-level and multi-angle technology integration, providing a solid technical guarantee for achieving safer and more efficient smart transportation.

[0054] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0056] Figure 1 It is a framework diagram of the system of the present invention;

[0057] Figure 2 Schematic diagram of the target appearance feature re-identification module structure. DETAILED DESCRIPTION

[0058] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0059] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0060] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0061] like Figure 1 As shown, the cross-domain multi-target tracking system and method for intersections based on C-V2X and radar-vision fusion proposed by the present invention includes a target detection module fused with radar and camera, a C-V2X perception compensation module, a target appearance feature re-identification module, a non-perception area trajectory estimation module and a cross-domain multi-target tracking module.

[0062] In one embodiment of the present invention, the target detection module is based on the YOLOv10 network, and the specific steps of the target detection module for radar and camera fusion to obtain the fusion target are:

[0063] S101. First, the camera and the radar need to be spatially synchronized, and the camera and the radar need to be jointly calibrated to determine the camera's internal and external parameter matrices and the conversion matrices among the world coordinate system, the camera coordinate system, the image coordinate system, the pixel coordinate system, and the radar coordinate system;

[0064] S102, synchronizing the camera and the radar by means of timestamp alignment. Since the operating frequencies of the camera and the radar are different, the sampling time of the radar is used as a reference to read the corresponding video stream data;

[0065] S103: For the target detected by the radar, its coordinates are converted to the pixel coordinate system of the camera, and a region of interest is dynamically constructed, including the width w, height h, and upper left corner coordinates (x 0 ,y 0 ) is determined by the following formula:

[0066]

[0067] Where f is the focal length of the camera, W is the actual width of the vehicle, and Y R is the distance between the target and the camera, A is the vehicle aspect ratio, A=1.3, (x, y) is the coordinate of the radar target;

[0068] S104. Calculate the intersection over union (IoU) (i, j) between the i-th region of interest and the j-th visual detection box, use 1-IoU (i, j) to construct a cost matrix, and input it into the Hungarian matching algorithm for bipartite graph optimal matching to complete the fusion of radar targets and visual targets.

[0069] In one embodiment of the present invention, a C-V2X perception compensation strategy is designed to solve the problem of target detection failure caused by roadside sensor failure due to weather, environment and other factors. The strategy is to integrate V2X communication to perform traffic object perception compensation in four situations according to whether there is a camera target or a radar target. The specific strategy is as follows:

[0070] There are camera targets and radar targets: match and fuse the camera targets and radar targets at the decision layer, and use BSM messages to further verify whether the fused targets are reasonable;

[0071] There is a camera target but no radar target: Use the vehicle position and speed parsed from the BSM message to construct a radar target and fuse it with the camera target;

[0072] No camera target but radar target: Match the vehicle position and speed parsed from the BSM message with the radar target, and then construct an adaptively sized target detection box in the pixel image based on the target size, type, and distance between the target and the camera to compensate for the camera target;

[0073] No camera target and no radar target: completely rely on V2X data for perception compensation, use BSM messages as radar targets, and compensate for camera targets using the compensation method for radar targets without camera targets.

[0074] In one embodiment of the present invention, the target appearance feature re-identification module structure is as follows: Figure 2 As shown in the figure, it consists of a global feature branch, a local feature branch and a multi-scale feature branch, and ResNet50 is selected as the Backbone network. After obtaining the global feature map, three feature extraction branches are used to extract feature vectors respectively, and the feature vectors are concatenated to obtain the final feature representation.

[0075] Among them, the global feature branch uses the spatial attention layer to assign different importances to different positions of the global feature map. The attention weight of a position depends on the total activation degree of all channels at that position. All learned attention weights compete with each other to highlight their importance, so that the model can cover more image details and has strong adaptability to vehicle images under different perspectives and backgrounds. Finally, the global feature map is optimized by triplet loss and ID loss.

[0076] The local feature branch divides the global feature map into multiple parts. Each part uses a channel attention layer to emphasize the most recognizable channel in the feature map, which can effectively suppress irrelevant features such as background and shadows and reduce noise interference. At the same time, the local features are weighted to increase the model's attention to the vehicle's significant information and suppress the interference of the sky area on the vehicle feature extraction. The weight parameters α=0.3 and β=0.4 are selected.

[0077] The multi-scale feature branch divides the global feature map into four parts and constructs a pyramid from coarse to fine to learn local features of different scales. The specific steps are as follows:

[0078] S201, number the four parts 1-4, and divide the top layer of the pyramid into two branches, corresponding to the two parts {1,2,3} and {2,3,4};

[0079] S202, the second layer of the pyramid is divided into three branches, corresponding to {1,2}, {2,3} and {3,4};

[0080] S203, the bottom layer of the pyramid is divided into four branches, corresponding to {1}, {2}, {3} and {4};

[0081] S204, performing global maximum pooling on each branch to capture the statistical characteristics of different channels of the sub-graph;

[0082] S205. Each branch applies a shared convolution filter with a filter kernel size of 1×1, converts it into a feature vector, and inputs it into the classifier.

[0083] In one embodiment of the present invention, in order to solve the problem that the visual information of the vehicle cannot be extracted in the blind spot of the field of vision, resulting in the interruption of the vehicle trajectory, an adaptive vehicle trajectory prediction method based on multi-model extended Kalman filtering is designed, specifically:

[0084] S301, using a multiple Kalman filter method instead of a complex model to cover most possible scenarios of vehicles in the blind spot, including four different extended Kalman filter models: a constant position model, a constant speed model, a constant acceleration model, and a vehicle turning model;

[0085] Among them, the constant position model is used to describe the state of a vehicle waiting for a traffic light, the constant speed model and the constant acceleration model are used to describe the state of a vehicle traveling in a straight line, and the vehicle turning model is used to describe the state of a vehicle turning.

[0086] S302. Analyze the BSM message in each time step to obtain the vehicle's motion state information, and select X as the vehicle's state vector:

[0087]

[0088] Where x and y are the position coordinates of the vehicle, v is the velocity, a is the acceleration, θ is the yaw angle, and ω is the angular velocity;

[0089] S303, constructing a state transfer matrix through the characteristics of different motion models, and initializing various parameters of the extended Kalman filter model;

[0090] S304, predefine a probability transfer matrix Π=[π ij ], where π ij Represents the probability of transitioning from model i to j, and sets the initial matrix to:

[0091]

[0092] Assume that the state probability vector of k at the current moment is μ(k), which represents the probability distribution of each model at the current moment:

[0093]

[0094] Using the Markov chain model, the state probability vector μ(k+1) at the next moment k+1 is calculated based on the current state probability vector and the transition probability matrix:

[0095] μ(k+1)=μ(k)Π

[0096] S305, estimating the trajectory and motion state of the vehicle according to the predicted model;

[0097] S306. Collect the model state transfer data at different time steps during the system operation, record the model actually used in each time step and the model of the next time step, and dynamically update the probability transfer matrix according to the observation data of the historical time step:

[0098] Assume that up to the current moment, the number of transitions from model i to model j is N ij , the formula for updating the probability transfer matrix Π is:

[0099]

[0100] In one embodiment of the present invention, target cross-domain data association uses a two-level matching strategy, which makes full use of the target appearance features, spatial features, and motion features, and uses an elliptical adaptive association gate to increase the accuracy of data association. The size and shape of the association gate are dynamically adjusted according to the speed and uncertainty of the fused target to adapt to the target's possible larger movement range and cover more potential positions. The specific matching strategy is:

[0101] S401, using the appearance feature distance of the target to build a cost matrix for primary matching, the appearance feature distances less than the threshold are determined as different targets, and an elliptical adaptive association gate is used to further screen targets that may be successfully matched;

[0102] S402, define the association gate as:

[0103]

[0104] Among them, (u 0 ,v 0 ) is the center of the ellipse, i.e., the pixel coordinate of the target, and a and b are the radii of the major and minor axes, respectively.

[0105] The dynamic adjustment strategy is:

[0106] Assume the target's velocity is v, split it into v x and v y , the greater the target speed, the longer the major axis of the associated gate should be to accommodate the target's possible larger movement range. The relationship between the adjustment of a and the speed is:

[0107]

[0108] Among them, a 0 is the radius of the major axis of the base, k v is the adjustment factor;

[0109] Using the covariance matrix Σ as the uncertainty matrix, the greater the uncertainty of the target, the wider the minor axis of the associated gate should be to cover more potential positions, and the relationship between the adjustment of b and the uncertainty is:

[0110]

[0111] Among them, b 0 is the basic minor axis radius, k Σ is the adjustment factor, λ max (∑) is the maximum eigenvalue of the covariance matrix, indicating the direction of maximum uncertainty;

[0112] S403, constructing an elliptical adaptive correlation gate with all detections as the center;

[0113] S404, assuming that the target is from the camera field of view V1 Move through the blind area to the camera field of view V 2 , project all predicted trajectories in the blind area to V 2 , perform secondary matching on the trajectories falling within the associated gate;

[0114] S405: Use the spatial features and motion features of the target to perform secondary matching and perform linear weighting, specifically:

[0115] Use the spatial position distance between the i-th detection and the j-th trajectory as the spatial feature similarity S p (i, j), using the speed difference between detection and trajectory as the motion feature similarity S v (i, j), the final association weight is:

[0116] W(i, j) = αS p (i, j)+(1-α)S v (i, j)

[0117] The cost matrix is ​​constructed with W(i, j) for secondary matching.

[0118] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A cross-domain multi-target tracking system at an intersection based on C-V2X and radar-vision fusion, characterized by: The system includes a target detection module, a C-V2X perception compensation module, a target appearance feature re-identification module, a trajectory estimation module, and a cross-domain multi-target tracking module; The target detection module obtains the appearance information, motion information and spatial information of the vehicle on the road based on the camera and radar, and performs coordinate conversion between world coordinates, camera coordinates, image coordinates and pixel coordinates to achieve the fusion of radar targets and visual targets; The C-V2X perception compensation module is used to perform perception compensation on the target detection module to increase the accuracy of fusion target detection; The target appearance feature re-identification module is used to extract appearance features from the appearance information obtained by the target detection module, and identify the target vehicle in the video sequences of multiple non-overlapping field of view cameras; The trajectory estimation module is used to estimate the trajectory and posture of the vehicle within the blind area of ​​the camera's field of view to ensure the consistency of the global ID of the tracked target; The cross-domain multi-target tracking module completes the multi-target tracking task under the single camera field of view according to the output of the target detection module and the C-V2X perception compensation module, and then combines the target appearance feature re-identification module and the trajectory estimation module to complete the data association and identity matching of the target trajectory under different fields of view, thereby realizing cross-domain multi-target tracking.

2. The cross-domain multi-target tracking system at an intersection according to claim 1, characterized in that: The target detection module acquires targets by integrating camera and radar based on the YOLOv10 network.

3. The cross-domain multi-target tracking system at an intersection according to claim 1, characterized in that: The C-V2X perception compensation module integrates V2X communication to perform perception compensation based on the target detection module to compensate for the target loss problem caused by the failure of the roadside sensor; The C-V2X perception compensation module performs perception compensation according to different situations of whether there is a camera target and whether there is a radar target.

4. The cross-domain multi-target tracking system at an intersection according to claim 1, characterized in that: The target appearance feature re-identification module includes a global feature branch, a local feature branch and a multi-scale feature branch. The feature vectors of the global feature map are extracted through the three feature extraction branches, and the feature vectors are then concatenated to obtain a feature representation.

5. The cross-domain multi-target tracking system at an intersection according to claim 1, characterized in that: The cross-domain multi-target tracking module is based on a cascade matching strategy of multi-source information fusion, fully utilizes the target appearance features, spatial features and motion features, and uses an elliptical adaptive association gate to increase the accuracy of data association. The size and shape of the association gate are dynamically adjusted according to the speed and uncertainty of the fused target to cover more potential positions of the target.

6. A cross-domain multi-target tracking method at an intersection based on C-V2X and radar-vision fusion, characterized in that: This method obtains radar targets and visual targets through radar and camera respectively, and fuses radar targets and visual targets to obtain appearance information, motion information and spatial information of vehicles on the road. C-V2X communication data is used to compensate for the perception of the fused detection targets of radar and camera to make up for the target detection failure caused by the failure of roadside sensors; Re-identify the appearance features from the appearance information of the vehicle, so as to identify the target vehicle in the video sequences of multiple non-overlapping field of view cameras; In the field of view of a single camera, local trajectories of vehicles in the field of view are predicted; for vehicles in non-camera blind spots, the local trajectories of the vehicles are predicted by a single-camera multi-target tracking algorithm; for vehicles in the blind spots of the camera, the local trajectories of the vehicles are predicted by an adaptive vehicle trajectory prediction method based on multi-model extended Kalman filtering, ensuring the consistency of the global ID of the tracked target; In the field of view of multiple different cameras, a cascade matching strategy based on multi-source information fusion is used to complete the data association and identity matching of target trajectories in different fields of view, thereby realizing cross-domain multi-target tracking. Among them, the appearance features, spatial features, and motion features of the vehicle are utilized, and an elliptical adaptive association gate is used to increase the accuracy of data association. At the same time, the size and shape of the association gate are dynamically adjusted according to the speed and uncertainty of the fused target to cover more potential positions of the target.

7. The method for tracking multiple targets across intersections according to claim 6, characterized in that: The perception compensation of the radar and camera fusion detection target through C-V2X communication data includes perception compensation according to different situations of whether there is a camera target and whether there is a radar target: When there are camera targets and radar targets, the camera targets and radar targets are matched and fused at the decision layer, and the BSM message is used to further verify whether the fused target is reasonable; When there is a camera target but no radar target, a radar target is constructed using the vehicle position and speed parsed from the BSM message and fused with the camera target; When there is no camera target but there is a radar target, the vehicle position and speed parsed from the BSM message are matched with the radar target, and then an adaptively sized target detection box is constructed in the pixel image according to the target size, type, and distance between the target and the camera to compensate for the camera target; When there is no camera target and no radar target, V2X data is relied upon for perception compensation, with BSM messages acting as radar targets, and the camera target is compensated using the compensation method used when there is no camera target but there is a radar target.

8. The method for tracking multiple targets across intersections according to claim 6, characterized in that: Extracting appearance features from the appearance information of the vehicle includes: extracting feature vectors of the global feature map through global feature branches, local feature branches, and multi-scale feature branches, and then concatenating the feature vectors to obtain feature representation, where: The global feature branch assigns different attention weights to different positions of the global feature map through a spatial attention layer, and the attention weight of each position depends on the total activation degree of all channels at that position; then the global feature map is optimized through triplet loss and ID loss; The local feature branch divides the global feature map into multiple parts, and each part uses a channel attention layer to emphasize the most recognizable channel in the feature map, thereby reducing noise interference; at the same time, the local features are weighted to increase attention to significant vehicle information and reduce noise interference; The multi-scale feature branch divides the global feature map into four parts, and constructs a pyramid from coarse to fine to learn local features of different scales.

9. The method for tracking multiple targets across intersections according to claim 6, characterized in that: The adaptive vehicle trajectory prediction method based on multi-model extended Kalman filtering includes: 1) Four extended Kalman filter models are used, including a constant position model, a constant speed model, a constant acceleration model, and a constant turning model; the constant position model is used to describe the state of a vehicle waiting for a traffic light, the constant speed model and the constant acceleration model are used to describe the state of a vehicle traveling in a straight line, and the constant turning model is used to describe the state of a vehicle turning; 2) Parse the BSM message in each time step to obtain the vehicle's motion state information, and select X as the vehicle's state vector: In the formula, x and y represent the position coordinates of the vehicle, v represents the speed, a represents the acceleration, θ represents the yaw angle, and ω represents the angular velocity; 3) Based on the characteristics of different motion models, the state transfer matrix is ​​constructed and the parameters of the extended Kalman filter model are initialized; 4) Predefine a probability transfer matrix Π = [π ij ], combining the observed data of the current time step to dynamically select the most appropriate model for the next prediction; π ij represents the probability of switching from model i to j; 5) Estimate the trajectory and motion state of the vehicle based on the predicted model; 6) Collect the model state transfer data at different time steps during system operation, record the model actually used in each time step and the model of the next time step, and dynamically update the probability transfer matrix based on the observation data of the historical time step.

10. The method for tracking multiple targets across intersections according to claim 6, characterized in that: The cascade matching strategy includes: 1) Use the appearance feature distance of the target to build a cost matrix for primary matching. Targets with appearance feature distances less than the threshold are judged as different targets, and an elliptical adaptive association gate is used to further screen targets that can be successfully matched; 2) Define the association gate as: Where (u0, v0) represents the center of the ellipse, i.e., the pixel coordinates of the target, and a and b represent the radii of the major and minor axes, respectively. The dynamic adjustment strategy of the association gate is: Assume the target's velocity is v, split it into v x and v y , the greater the target speed, the longer the major axis of the associated gate should be to accommodate the target's potential larger movement range. The relationship between the adjustment of a and speed is: In the formula, a0 represents the radius of the major axis of the foundation, k v represents the adjustment factor; Using the covariance matrix Σ as the uncertainty matrix, the greater the uncertainty of the target, the wider the minor axis of the associated gate should be to cover more potential positions. The relationship between the adjustment of b and the uncertainty is: Where b0 represents the radius of the minor axis of the foundation, k ∑ represents the adjustment factor, λ max (Σ) represents the maximum eigenvalue of the covariance matrix, indicating the direction of maximum uncertainty; 3) Construct an elliptical adaptive correlation gate centered on all detections; 4) Assuming that the target moves from the camera field of view V1 to the camera field of view V2 through the blind spot, all predicted trajectories in the blind spot are projected to V2, and secondary matching is performed for the trajectories that fall within the associated gate; 5) Use the spatial features and motion features of the target for secondary matching and perform linear weighting: The spatial position distance between the i-th detection and the j-th trajectory is used as the spatial feature similarity, the speed difference between the detection and the trajectory is used as the motion feature similarity, and the spatial feature similarity and the motion feature similarity are linearly weighted to construct a cost matrix for secondary matching.

Citation Information

Patent Citations

  • Test method, device and equipment of vehicle and road cloud sensing system and storage medium

    CN114323693A

  • Vehicle-road cooperation multi-target tracking method and system

    CN115577315A

  • Intersection multi-target cross-domain tracking method based on overlapped view

    CN116894855A

  • Radar and camera fusion based wireless communication misbehavior detection

    US20240034337A1

Cited By

  • Top-crossing tracking processing method and device, equipment, storage medium and computer product

    CN120070508A

  • Method and system for industrially controlling material handling vehicle

    CN120848323A

  • Traffic data base construction and safety management method and system for vehicle-road cooperation

    CN121305923A

  • Automatic driving decision-making method and system based on multi-source perception

    CN121316910A

  • Traffic intersection tracking method

    CN121391923A