A cross-domain tracking method for multiple targets at intersections based on overlapping fields of view

Abstract

The present invention relates to a multi-target cross-domain tracking method at an intersection based on overlapping vision, which belongs to the field of vehicle-road collaboration and intelligent transportation, including a radar and visual fusion target detection model, a target appearance feature re-identification model, an overlapping area coordinate mapping model, a neighborhood trajectory priority matching model based on target movement features, and a multi-level data association model of multi-source information fusion. The intelligent roadside multi-source information acquisition equipment and the target movement law are used to reduce the problems of inter-class similarity and intra-class difference in the re-identification field, and the deep learning and adaptive vision scene two-stage training strategy are used to improve the recognition of re-identification features and the accuracy of cross-domain target tracking, complete the accurate perception of traffic objects at smart intersections, and obtain the global motion trajectory of traffic objects at intersections. The tracking results are sent to the cloud and the roadside unit RSU, and the vehicle-mounted unit OBU is sent through the V2X communication technology to make up for the visual field blind spot problem existing in single perception.

Description

Technical Field

[0001] The present invention belongs to the field of vehicle-road collaboration and intelligent transportation, and relates to a multi-target cross-domain tracking method at an intersection based on overlapping fields of view. Background Art

[0002] The development of vehicle-road collaboration based on V2X communication has been greatly promoted in recent years, and the development of traffic intersections has gradually shifted to the development of smart intersections with multi-faceted collaboration of "people-vehicle-road-cloud", becoming an important part of future intelligent transportation. In the intelligent transportation system, environmental perception is a prerequisite for the entire system. Its target perception accuracy and perception continuity determine the functional perfection of subsequent subsystems. Therefore, environmental perception at traffic intersections plays a vital role. Surveillance cameras and radars and other environmental perception devices deployed at traffic intersections are gradually covering all intersections. With the development of computer vision and deep learning, environmental perception algorithms based on large amounts of data input such as video streams have gradually been applied. Among them, the target tracking algorithm is another major environmental perception algorithm that has been widely used in the field of intelligent transportation after the target detection algorithm.

[0003] Initially, target tracking was mainly aimed at tracking targets under a single camera. However, as the role of target tracking in the field of intelligent transportation increases and as the perception area expands, the field of view coverage of a single camera often cannot meet the tracking of the global trajectory of the target. This requires the target to drive out of the current field of view and continue to be tracked after it appears in another field of view. Therefore, cross-domain multi-target tracking came into being. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a cross-domain tracking method for multiple targets at an intersection based on overlapping fields of view.

[0005] In order to achieve the above object, the present invention provides the following technical solutions:

[0006] A method for tracking multiple targets across multiple domains at an intersection based on overlapping fields of view comprises the following steps:

[0007] S1: Detect traffic objects in different areas through radar and vision fusion target detection model, providing accurate and reliable target information for tracking algorithm;

[0008] S2: Provides the appearance features of the target in different fields of view and at various angles through the graph-embedded domain adaptive object re-identification model;

[0009] S3: Mapping the targets at the edges of adjacent fields of view into overlapping areas through the overlapping area coordinate mapping model, and mapping the targets in the previous field of view to the next field of view;

[0010] S4: A neighborhood trajectory priority matching model based on target movement characteristics divides the target trajectory into regions according to the spatiotemporal constraints, pre-filters the trajectories that do not meet the current target matching object, and prioritizes the trajectories that meet the requirements in the trajectory matching stage;

[0011] S5: A multi-level data association model based on multi-source information fusion uses the characteristics of trajectory and target time flow data, and uses the target's appearance features, local spatial position, and global spatial position to build a three-level matching algorithm to improve the success rate of data association;

[0012] S6: After the data association is completed, the data of each successfully matched trajectory is updated to obtain the information of the target at the current moment, and the target and trajectory at the next moment are matched based on this information;

[0013] S7: Deploy the model in practice.

[0014] Further, step S1 specifically includes the following steps:

[0015] S11: Before radar-vision fusion, perform spatiotemporal synchronization of radar and vision sensors, that is, perform joint calibration in space and time alignment in time; select the world coordinate system as a bridge to jointly calibrate the radar coordinate system and the camera coordinate system; obtain the rotation angles of the three axes from the radar's external parameter matrix, set the corresponding α, β, and γ as the rotation angles of the Z axis, Y axis, and X axis, multiply each component by the corresponding rotation angle, and add the corresponding translation matrix to complete the conversion between the radar and the world coordinate system. The final conversion formula is as follows:

[0016]

[0017] The camera coordinate system also needs to establish a conversion relationship with the world coordinate system; the camera's extrinsic parameter matrix Rc is obtained through the camera extrinsic parameter calibration process, and the conversion formula between the camera and the world coordinate system is as follows:

[0018]

[0019] Then convert the camera coordinate system to the pixel coordinate system. Finally, the conversion relationship between the radar coordinate system and the pixel coordinate system is expressed as:

[0020]

[0021] Where X r , Y r , Z r is the radar coordinate, R is the transfer matrix between the radar coordinate system and the camera coordinate system, f is the focal length of the camera, dx and dy represent the actual length of a single pixel of the camera on the imaging sensor in the horizontal and vertical directions, and u0 and v0 represent the center position of the image;

[0022] S12: Time-align the two sensors, use the timestamp of the point cloud data collected by the radar as a reference, and read the corresponding image data;

[0023] S13: The data set is enhanced by a multi-target geometric fusion data enhancement method. The complete target is randomly selected to overlap other targets in geometric areas, the target itself is erased using geometric figures, the erased parts are filled with parts of other targets and marked as occluded targets to simulate the situation where the target is blocked by a large area. Small targets, incomplete targets at the edge of the field of view, small targets and occluded targets are added to the data set. The processed data set is used to train the radar and vision fusion target detection model.

[0024] Furthermore, the graph-embedded domain-adaptive object re-identification model in step S2 is specifically as follows:

[0025] First, we improve the model based on Vision Transformer. We use a combination of CNN and Transformer as the feature extraction network for target re-identification. We use ResNeXt network as the PatchEmbedding module of Vision Transformer. We use Instance Normalization and Batch Normalization parallel structures to replace the original single normalization layer, which improves the model's ability to extract the overall appearance features of the target.

[0026] After the image passes through the CNN convolution model to output the global appearance features, the original image is fragmented into N patches and input into the Patch Embedding layer for initial feature extraction. The position encoding patch and the camera feature offset encoding patch are then fused and input into the Transformer Layer to learn the local features of the target in each field of view.

[0027] Through the graph convolutional local feature fusion model, N patches are locally spatially displaced and their positions are recorded in groups. The central patch of each group is used as a node to construct a spatial similarity graph. The connection relationship of each patch is calculated by graph convolution to better integrate the local features and global features of each target.

[0028] Furthermore, the graph convolutional local feature fusion model is:

[0029] The local features of the target’s N patches are further divided according to the step size s, and the division results are regarded as graph nodes. The distance between each node is expressed as:

[0030]

[0031] in represents the strength of the association between nodes i and j, v i represents the feature of node i, F Cosine represents the spatial cosine distance, expressed as Use the spatial position and its own features to build the adjacency matrix and degree matrix, complete the construction of the graph neural network, and perform graph convolution training. After continuous iteration, the adjacency matrix and degree matrix are updated to improve the connection and distinction between the local features of the target, and improve the discrimination of the local feature differences of the target by the graph embedded field adaptive target re-identification model; the update rules between each node are expressed as:

[0032]

[0033] Among them U i is represented as node features, L is represented as the position relationship matrix between nodes, D is represented as the degree matrix, A is represented as the adjacency matrix, X i Represented as the i-th local appearance feature, Represents a learnable parameter matrix used to learn local features x i The weight of the feature in the total features;

[0034] The loss function uses the Triplet loss triplet loss function to calculate the inter-class difference and intra-class similarity loss, and the cross entropy loss is used as the classification loss; three targets are selected as a group to calculate the loss, A1, A2, and A3 form a group of samples, where A1 and A2 are the same target, but the appearance difference is large, simulating the intra-class difference; A3 and A1 and A2 belong to different targets, but the appearance difference is small, simulating the inter-class similarity; the loss function corresponding to the triplet is as follows:

[0035]

[0036] Where f(x) is the target feature extraction network, are the three targets in the triplet, α is a custom distance parameter, which indicates the minimum distance between positive and negative sample pairs.

[0037] Furthermore, the training process of the graph embedded domain adaptive object re-identification model is as follows:

[0038] S101. Determine that the training epoch number of the deep learning model is 80; determine the learning rate of the model training, the initial learning rate is set at 0.01, and use the warmup strategy. The learning rate increases in the first 10 epochs, and the learning rate in subsequent rounds gradually decays according to the cosine decay ratio;

[0039] S102. Select SGD optimizer as the model training optimizer, and select triple loss function as the loss function of the model;

[0040] S103. Determine that the dimension of the target feature vector finally output by the model is 756;

[0041] S104. The initial values ​​of the position code and the camera bias code are encoded using all zeros;

[0042] S105. The stride of the last convolutional layer of the Patch Embedding module is 16*16;

[0043] S106. Set Batch Size to 32;

[0044] S107. Start training.

[0045] Furthermore, the image embedded domain adaptive object re-identification model is trained in two stages through the adaptive field of view scene two-stage training strategy. During the adaptive domain training process, the re-identification model trained in the first stage is used to extract its features g for each target. I , then extract all the target features at the current moment captured by the target lens and take the feature average The characteristic average value Acting on the target feature g I Then extract the features of the target in all shots. I , and find the mean, and set the feature t I , feature g I and Features The final target feature is formed by fusion according to different weights; based on this feature, the original training model is adjusted. The core formula for the second stage training of the model is as follows.

[0046]

[0047]

[0048] The eigenvalue f I That is, the target appearance feature that integrates the camera field of view feature. The final fusion feature is a fusion of the features of each angle of the target and each camera field of view;

[0049] The two-stage training process is:

[0050] S201. Collect pictures in the field deployment scene, build a small data set, and use the target detection model to perform target capture;

[0051] S202. Grouping the captured images according to the target;

[0052] S203. Grouping the captured images by camera;

[0053] S204. Set Epoch to 5, the initial learning rate to 0.001, and adopt the learning rate decay strategy in each round;

[0054] S205. Start training;

[0055] S206. Extract features of all targets under camera i using the model trained in the first stage, and calculate the average value to obtain the feature vector

[0056] S207. Traverse each single target image to obtain the target appearance feature vector g I , according to the formula Under the action of weight α, we can get f I ;

[0057] S208. Traverse all images of each target under all lenses to obtain the feature vector t I , according to the formula The final target appearance is obtained under the action of weight β

[0058] S209. I and Perform loss calculation, perform backpropagation, and fine-tune the model.

[0059] Further, step S3 specifically includes: finding four or more groups of points at the same position in the overlapping area in advance, calculating the homography matrix H representing the mapping relationship between the two fields of view, and using the homography matrix to construct the mapping relationship between the overlapping areas of the two fields of view; the target coordinate transformation relationship in the two fields of view is shown in the following formula:

[0060]

[0061] is the coordinate of the target center point in the overlapping area of ​​the previous field of view. After the transformation of the coordinate mapping relationship matrix H, the coordinate in the next field of view is obtained.

[0062] Further, step S4 specifically includes the following steps:

[0063] S41: First, use the movement direction of the target detected by the radar to filter the trajectory when matching the trajectory. When it is determined that there is no turn, the target with the same movement direction is matched first.

[0064] S42: filtering according to the moving areas adjacent to the target, filtering the trajectory in the area that the target cannot reach in a short time; for each camera coverage area, demarcating the area according to traffic regulations, and indicating the area where the target can move continuously.

[0065] Furthermore, in step S5, before matching the target with the trajectory, the Kalman filter is used to predict the position of the trajectory at the previous moment at the current moment, and the position information is assigned to the trajectory; two different motion models are established for the target motion state, namely, a constant rate motion model and a uniform linear motion model, and the model parameters are the lateral position, longitudinal position and their respective first-order derivatives of the target. The other model is a constant turn rate and speed model, and the model parameters include the target's lateral and longitudinal positions and steering angles as well as their respective first-order derivatives. The two models are adaptively selected by the interactive multi-model Kalman filter at different stages according to the target's motion state to predict and update the target's motion state.

[0066] Further, step S6 specifically includes: firstly, using the Mahalanobis space distance between the trajectory and the target appearance feature vector to construct a cost matrix, performing the first-level matching between the target and the trajectory, directly entering the second-level matching for the target and trajectory that failed to match, and then judging whether their field of view IDs are the same as the successfully matched target and trajectory, if they are the same, it indicates that the target has not crossed the field of view, and entering the second-level matching for auxiliary matching, if the field of view IDs are not the same, it indicates that the target has crossed the domain, and at this time, the cost matrix is ​​calculated using the IOU of the target's mapping bounding box and the bounding box of the trajectory for matching;

[0067] If both the first and second level matches are successful, the target is deemed to be matched with the track successfully, and data association can be performed to update the track with the target information at the current moment;

[0068] If the match fails, the third level matching is entered, and the state vector is constructed using the absolute position information of the latitude and longitude of the target and the trajectory, the target speed, and the driving angle to calculate the cost matrix for matching;

[0069] The data association module of the target and trajectory is constructed through three-level matching.

[0070] The beneficial effect of the present invention is that compared with the prior art, the present invention makes full use of the intelligent roadside multi-source information collection equipment and the target movement law to reduce the problems of inter-class similarity and intra-class difference in the re-identification field, and at the same time uses deep learning and adaptive vision scene two-stage training strategy to improve the recognition of re-identification features, which can effectively improve the accuracy of cross-domain tracking of targets, complete the precise perception of traffic objects at smart intersections, and obtain the global movement trajectory of traffic objects at intersections. At the same time, the tracking results can be sent to the cloud and the roadside unit RSU, and sent to the on-board unit OBU through V2X communication technology, which can effectively make up for the blind spot problem of single perception.

[0071] Other advantages, objectives and features of the present invention will be described in the following description to some extent, and to some extent, will be obvious to those skilled in the art based on the following examination and study, or can be taught from the practice of the present invention. The objectives and other advantages of the present invention can be realized and obtained through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0072] In order to make the purpose, technical solutions and advantages of the present invention more clear, the present invention will be described in detail below in conjunction with the accompanying drawings, wherein:

[0073] Figure 1 This is a system framework diagram of the present invention;

[0074] Figure 2 This is a structural diagram of the re-identification model of the present invention;

[0075] Figure 3 This is a diagram of the two-stage training process of the re-identification model of the present invention;

[0076] Figure 4 The target movable area planning map of the intersection of the present invention;

[0077] Figure 5 The target motion state model diagram of the present invention, wherein (a) is a constant rate motion model diagram, and (b) is a constant rotation rate and speed model diagram;

[0078] Figure 6 This is a flow chart of the multi-level matching algorithm of the target and trajectory of the present invention. DETAILED DESCRIPTION

[0079] The following describes the embodiments of the present invention by specific examples, and those skilled in the art can easily understand other advantages and effects of the present invention from the contents disclosed in this specification. The present invention can also be implemented or applied through other different specific embodiments, and the details in this specification can also be modified or changed in various ways based on different viewpoints and applications without departing from the spirit of the present invention. It should be noted that the illustrations provided in the following embodiments only illustrate the basic concept of the present invention in a schematic manner, and the following embodiments and features in the embodiments can be combined with each other without conflict.

[0080] Among them, the drawings are only used for illustrative explanations, and they only represent schematic diagrams rather than actual pictures, and should not be understood as limitations on the present invention. In order to better illustrate the embodiments of the present invention, some parts of the drawings may be omitted, enlarged or reduced, and do not represent the size of actual products. For those skilled in the art, it is understandable that some well-known structures and their descriptions in the drawings may be omitted.

[0081] The same or similar numbers in the drawings of the embodiments of the present invention correspond to the same or similar parts; in the description of the present invention, it should be understood that if the terms "upper", "lower", "left", "right", "front", "rear", etc. indicate the orientation or position relationship, they are based on the orientation or position relationship shown in the drawings, which is only for the convenience of describing the present invention and simplifying the description, rather than indicating or implying that the device or element referred to must have a specific orientation, be constructed and operate in a specific orientation. Therefore, the terms describing the position relationship in the drawings are only used for illustrative purposes and cannot be understood as limiting the present invention. For ordinary technicians in this field, the specific meanings of the above terms can be understood according to specific circumstances.

[0082] A method and system for tracking multiple targets across intersections based on overlapping fields of view. Figure 1As shown in the figure, it includes radar and vision fusion target detection model, graph embedded domain adaptive target re-identification model, target space coordinate mapping model under overlapping field of view, neighborhood trajectory priority matching model based on target movement characteristics, and multi-level data association model of multi-source information fusion. Among them, the radar and vision fusion target detection model uses vision and radar to detect targets on the same road section at the same time, and the results are associated with the Hungarian matching algorithm, so that the target has both visual and radar data information. The target appearance feature re-identification model extracts the target from the picture, obtains the appearance features of the target, and uses a 756-dimensional high-dimensional vector representation to give the target appearance feature information. The overlapping area coordinate mapping model maps the targets in the overlapping area at the intersection of the field of view to achieve target transfer between fields of view. The neighborhood trajectory priority matching model based on target movement characteristics uses the target movement law and traffic rules to construct a spatiotemporal constraint relationship, filters the target trajectory between regions, and prioritizes the matching of trajectories and targets that meet the movement law. The multi-level data association model of multi-source information fusion performs multi-level cascade matching on targets and trajectories that meet the requirements, and performs data association on targets and trajectories that successfully match.

[0083] In the target detection stage, radar and vision fusion are used for target detection. The specific detection process includes the following stages:

[0084] The stage of spatiotemporal synchronization between radar and camera. Before radar-vision fusion, it is necessary to synchronize the radar and vision sensors in spatiotemporal time, that is, to perform joint calibration in space and time alignment in time. Different sensors have their own coordinate systems, and the transmitted coordinate origins are inconsistent, so the world coordinate system is used as a bridge to jointly calibrate the radar coordinate system and the camera coordinate system. Obtain the rotation angles of the three axes from the radar's external parameter matrix, set the corresponding α, β, and γ as the rotation angles of the Z axis, Y axis, and X axis, multiply each component by the corresponding rotation angle, and add the corresponding translation matrix to complete the conversion between the radar and the world coordinate system. The final conversion formula is as follows:

[0085]

[0086] The camera coordinate system also needs to establish a conversion relationship with the world coordinate system. After the camera external parameter calibration process, the camera's external parameter matrix Rc is obtained. The conversion formula between the camera and the world coordinate system is as follows:

[0087]

[0088] Then convert the camera coordinate system to the pixel coordinate system. Finally, the conversion relationship between the radar coordinate system and the pixel coordinate system is expressed as:

[0089]

[0090] Where X r , Y r , Z r is the radar coordinate, R is the transfer matrix between the radar coordinate system and the camera coordinate system, f is the focal length of the camera, dx and dy represent the actual length of a single pixel of the camera on the imaging sensor in the horizontal and vertical directions, and u0 and v0 represent the center position of the image.

[0091] After the radar camera is calibrated together, the two sensors need to be time-aligned. Since the operating frequencies of the camera and radar are different, the image information and point cloud data will not match, so a time reference needs to be specified and the same timestamp needs to be used. This paper uses the timestamp alignment method, using the timestamp of the point cloud data collected by the radar as the reference to read the corresponding image data.

[0092] During the training phase of the visual target detection model, the target detection model uses DETR as the training model and BDD100K as the main training data set. However, considering that cross-domain target tracking needs to pay attention to the detection accuracy of the target at the boundary of the field of view, a multi-target geometric fusion data enhancement method is designed and proposed, and the data set is enhanced using this data enhancement method. The complete target is randomly selected to overlap and splice with other targets in geometric areas. The target itself is now erased using geometric figures, and the erased part is filled with part of the area of ​​other targets and marked as an occluded target to simulate the situation where the target is blocked by a large area, which is convenient for improving the model's attention extraction. In addition, small targets, incomplete targets at the edge of the field of view, small targets, and occluded targets are added to the data set to improve the generalization ability of the model.

[0093] In the target re-identification stage, a graph-embedded domain-adaptive target re-identification model is designed and proposed to improve the fusion effect of the target's global features and local features and the recognition of all angles. The model uses Vision Transformer as the backbone network to improve the model and build a re-identification model. First, the model is improved based on Vision Transformer, and a combination of CNN and Transformer is used as the feature extraction network for target re-identification. The ResNeXt network is used as the Patch Embedding module of Vision Transformer. The Instance Normalization and Batch Normalization parallel structures are used to replace the original single normalization layer to improve the model's ability to extract the overall appearance features of the target. Figure 2As shown in the figure, after the image passes through the CNN convolution model to output the global appearance features, the original image is fragmented into N patches, input into the Patch Embedding layer for initial feature extraction, and then fused with the position encoding patch and the camera feature bias encoding patch, and input into the Transformer Layer to learn the local features of the target in each field of view. In order to improve the fit between global features and local features, a graph convolution local feature fusion algorithm is designed and proposed. The N patches are locally spatially displaced and the positions are recorded in groups. The central patch of each group is used as a node to construct a spatial similarity graph. The connection relationship of each patch is calculated by graph convolution to better fuse the local features and global features of each target. The design process of the graph convolutional neural network is as follows: the local features of the N patches of the target are further divided according to the step size s, and the results of the division are regarded as graph nodes. The distance between each node is expressed as:

[0094]

[0095] in represents the strength of the association between nodes i and j, v i represents the feature of node i, F Cosine represents the spatial cosine distance, expressed as Use the spatial position and its own features to build the adjacency matrix and degree matrix, complete the construction of the graph neural network, and perform graph convolution training. After continuous iteration, the adjacency matrix and degree matrix are updated to improve the connection and distinction between the local features of the target, and improve the re-identification model's ability to distinguish the differences in the local features of the target. The update rules between each node are expressed as:

[0096]

[0097] Among them U i is represented as node features, L is represented as the position relationship matrix between nodes, D is represented as the degree matrix, A is represented as the adjacency matrix, X i Represented as the i-th local appearance feature, Represents a learnable parameter matrix used to learn local features x i The weight of the feature in the total features.

[0098] The loss function uses the triplet loss function to calculate the inter-class difference and intra-class similarity loss, and the cross entropy loss is used as the classification loss. Three targets are selected as a group to calculate the loss. A1, A2, and A3 form a group of samples, where A1 and A2 are the same target, but the appearance difference is large, simulating the intra-class difference; A3 and A1 and A2 belong to different targets, but the appearance difference is small, simulating the inter-class similarity. The loss function corresponding to the triplet is as follows:

[0099]

[0100] Where f(x) is the target feature extraction network, are the three targets in the triplet, and α is a custom distance parameter, which indicates the minimum distance between positive and negative sample pairs. The significance of this loss function is to limit the difference between the same target and the difference between different targets to the interval with the hyperparameter ε as the threshold. As the model is trained, the loss continues to decrease, gradually reducing the similarity between classes and the difference within classes.

[0101] The training process of this stage is as follows:

[0102] S101. Determine that the training epoch number of the deep learning model is 80; determine the learning rate of the model training, with the initial learning rate set at 0.01. Use the warmup strategy, with the learning rate increasing in the first 10 epochs and then gradually decaying in subsequent rounds according to the cosine decay ratio.

[0103] S102. Select the SGD optimizer as the model training optimizer, and select the triplet loss function as the loss function of the model.

[0104] S103. Determine that the dimension of the target feature vector finally output by the model is 756.

[0105] S104. The initial values ​​of the position code and the camera offset code use all-zero code.

[0106] The stride of the last convolutional layer of the S105.Patch Embedding module is 16*16.

[0107] S106. Set Batch Size to 32.

[0108] S107. Start training.

[0109] Secondly, in order to solve the problem that the re-identification model training scene is quite different from the application scene, a two-stage training strategy for adaptive field of view is designed. As shown in the figure, during the adaptive field training process, the re-identification model trained in the first stage is used to extract its features g for each target. I , then extract all the target features at the current moment captured by the target lens and take the feature average To reduce the camera bias between cross-camera images, the feature average value Acting on the target feature g I Then extract the features of the target in all shots. I , and find the mean, and set the feature t I , feature g I and Features The final target feature is formed by fusing different weights. Based on this feature, the original training model is adjusted. The core formula of the second stage training of the model is as follows.

[0110]

[0111]

[0112] The eigenvalue f I That is, the target appearance feature that integrates the camera field of view feature. It is the final fusion feature that integrates the features of each angle of the target and each camera field of view.

[0113] like Figure 3 As shown in the figure, the two-stage training process of the re-identification model is:

[0114] S201. Collect pictures in field deployment scenarios, build small datasets, and use target detection models for target capture.

[0115] S202. Group the captured images according to the targets.

[0116] S203: Group the captured images according to the camera.

[0117] S204. Set Epoch to 5, the initial learning rate to 0.001, and adopt the learning rate decay strategy in each round.

[0118] S205. Start training.

[0119] S206. Extract features of all targets under camera i using the model trained in the first stage, and calculate the average value to obtain the feature vector

[0120] S207. Traverse each single target image to obtain the target appearance feature vector g I , according to the formula Under the action of weight α, we can get f I .

[0121] S208. Traverse all images of each target under all lenses to obtain the feature vector t I , according to the formula The final target appearance is obtained under the action of weight β

[0122] S209. I and Perform loss calculation, perform backpropagation, and fine-tune the model.

[0123] In the coordinate mapping stage, a spatial mapping strategy under overlapping fields of view is designed to address the problem of target loss caused by target coordinate system jumps at the intersection of fields of view. When the target drives into the overlapping area at the edge of the field of view, in order to better transfer target information between fields of view, the target in the previous shot of the overlapping area is mapped downward and matched with the same target in the next field of view to complete the cross-domain matching of the target. It is necessary to find four or more groups of points at the same position in the overlapping area in advance, calculate the homography matrix H representing the mapping relationship between the two fields of view, and use the homography matrix to construct the mapping relationship between the overlapping areas of the two fields of view. The target coordinate transformation relationship in the two fields of view is shown in the following formula:

[0124]

[0125] is the coordinate of the target center point in the overlapping area of ​​the previous field of view. After the transformation of the coordinate mapping relationship matrix H, the coordinate in the next field of view can be obtained.

[0126] In the trajectory filtering stage, in order to solve the problem of high similarity between traffic object classes and high difference within a class, a neighborhood trajectory priority matching strategy based on target movement characteristics is designed. First, the movement direction of the target detected by radar is used to filter the trajectory when matching. When it is judged that there is no turn, the target with the same movement direction is matched first. Secondly, it is filtered according to the moving area adjacent to the target. Figure 4 As shown in the figure, the target movable area is divided into 5 areas in total, and the target continuous moving area connection lines include the following: ①—>④—>③, ①—>④—>⑤, ③—>④—>②, ③—>④—>⑤, ⑤—>④—>②, ⑤—>④—>③. When matching the trajectory of the target, filter the trajectories in the area that the target cannot reach in a short time. For each camera coverage area, the area is demarcated according to traffic rules, and the target continuous moving area is indicated. For example, the target in area ③ can only come from ④, so when matching the trajectory of the target in area ③ in the field of view, the trajectory in area ④ will be matched first. The trajectory filtering strategy can greatly reduce the number of trajectories for each target matching, and greatly reduce the probability of similarity between classes. It not only reduces the amount of calculation, but also improves the matching accuracy.

[0127] In the target and trajectory data association stage, in order to solve the problem of data association failure caused by the high dependence on single trajectory matching of information sources, a multi-level data association strategy with multi-source information fusion is designed. The cost matrix is ​​constructed using the target's appearance features, detection box intersection ratio, longitude and latitude, and other information. A cascade matching algorithm is designed according to the regional trajectory filtering requirements to complete the data association between the current target and the historical trajectory. Before matching the target with the trajectory, it is necessary to use Kalman filtering to predict the position of the trajectory at the previous moment at the current moment, and assign the position information to the trajectory. This requires modeling the target's motion state. Figure 5 As shown in (a)-(b) in the figure, considering that the target has two choices at the intersection, namely, going straight or turning, two different motion models are established for the target motion state, namely, the constant rate motion model and the uniform linear motion model. The model parameters are the lateral position, longitudinal position and their respective first-order derivatives of the target. The other model is a constant turn rate and speed model, and the model parameters include the target's lateral and longitudinal positions and steering angles as well as their respective first-order derivatives. The two models are adaptively selected by the interactive multi-model Kalman filter at different stages according to the target's motion state to predict and update the target's motion state.

[0128] After the trajectory prediction stage is completed, the trajectories are grouped according to their field of view labels and the areas they are located in, and the trajectories are grouped according to the trajectory filtering strategy, and matched with the current target according to the priority. Figure 6 As shown in the figure, the cost matrix is ​​first constructed by using the Mahalanobis space distance between the trajectory and the target appearance feature vector to perform the first-level matching between the target and the trajectory. For the target and trajectory that failed to match, the second-level matching is directly entered. The target and trajectory that successfully matched then determine whether their field of view IDs are the same. If they are the same, it means that the target did not cross the field of view, and the second-level matching is performed for auxiliary matching. If the field of view IDs are different, it means that the target has moved across the domain. At this time, the cost matrix is ​​calculated using the iou of the target's mapping bounding box and the trajectory's bounding box for matching. If both the first and second level matching are successful, it can be determined that the target and the trajectory are successfully matched, and data association can be performed, and the target information at the current moment is updated to the trajectory; if the match is not successful, the third level matching is entered, and the state vector is constructed using the absolute position information of the latitude and longitude of the target and the trajectory, the target speed, the driving angle and other information to calculate the cost matrix for matching. The three-level matching constitutes the data association module between the target and the trajectory.

[0129] The actual deployment stage can be roughly divided into two steps, namely, accurate perception and data fusion of traffic objects, and cross-domain data association of targets and trajectories. In the accurate perception of traffic objects stage, first, a set of cameras and millimeter-wave radars are installed on the four directions of the intersection, and joint calibration and time synchronization are performed. Real-time detection and data fusion are performed on traffic objects in the same direction in terms of vision and radar, and the target re-identification model is used to extract the appearance features of the target, and the current detection target Detections are constructed. The fusion and filtering of vision and radar ensure the accuracy of target detection, providing accurate input for the subsequent target tracking algorithm. In the cross-domain data association stage of targets and trajectories, the key coordinate points of the overlapping areas between each field of view are firstly marked to obtain the mapping relationship between each field of view. After the target drives to the overlapping area, it can be accurately mapped and the Detections are updated. Before the track Tracks are matched with the target Detections, in order to ensure the accuracy of the track position, the Kalman filter algorithm is first used to predict the current position of the track, and then the appearance features and spatial distances are multi-level matched with the Detections. To ensure the accuracy of data association, for trajectories that failed to match, if they are within the survival period, they will continue to participate in the matching at the next moment. The specific process is as follows:

[0130] S301. Install the camera and millimeter-wave radar, and perform joint calibration and time-space calibration.

[0131] S302. Perform target detection, capture the visual target detection results in the original video image, and input them into the ReID re-identification model to extract the target appearance features.

[0132] S303. Perform decision-making data fusion on the visual target and the radar target, calculate the IOU based on the bounding boxes of the visual target and the radar target, perform Hungarian matching, and construct Detections by obtaining the fused target.

[0133] S304. Coordinate mapping is performed for targets existing in the overlapping area of ​​the fields of view. The targets in field of view i are mapped to field of view i-1, and data is fused with the corresponding targets in field of view i-1 to complete the data transfer of target data at the boundary of fields of view, and the global detections are updated.

[0134] S305. The trajectories in the trajectory pool are predicted according to the motion states of different trajectories using interactive multi-model Kalman filtering to obtain the detection box position of the trajectory at the previous moment at the current moment.

[0135] S306. Group all tracks according to the field of view id.

[0136] S307. Match the tracks Tracks and target Detections according to the groups.

[0137] S308. Determine the regional location of each target, filter the trajectories according to the neighborhood space-time constraint rules, and give priority to matching trajectories that conform to the regional motion rules.

[0138] S309. Use the target appearance features and absolute space features to construct a cost matrix for multi-level cascade matching.

[0139] S310. Match the successful detections and tracks, update the target information to the track, and complete data association and transmission.

[0140] S311. Tracks that failed to match are deleted if they exceed their life cycle.

[0141] S312. Detections that failed to match indicate that a new target has appeared in the field of view, and the trajectory is initialized.

[0142] Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of the present invention rather than to limit it. Although the present invention has been described in detail with reference to the preferred embodiments, those skilled in the art should understand that the technical solution of the present invention can be modified or replaced by equivalents without departing from the purpose and scope of the technical solution, which should be included in the scope of the claims of the present invention.

Claims

1. A method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view, characterized in that: The following steps are involved: S1: Detect traffic objects in different areas through radar and vision fusion target detection model, providing accurate and reliable target information for tracking algorithm; S2: The appearance features of the target in different fields of view and at various angles are provided by a graph-embedded domain-adaptive target re-identification model; the graph-embedded domain-adaptive target re-identification model is specifically as follows: First, we improve the model based on Vision Transformer. We use a combination of CNN and Transformer as the feature extraction network for target re-identification. We use ResNeXt network as the PatchEmbedding module of Vision Transformer. We use Instance Normalization and Batch Normalization parallel structures to replace the original single normalization layer, which improves the model's ability to extract the overall appearance features of the target. After the image passes through the CNN convolution model to output the global appearance features, the original image is fragmented into N patches and input into the Patch Embedding layer for initial feature extraction. The position encoding patch and the camera feature offset encoding patch are then fused and input into the Transformer Layer to learn the local features of the target in each field of view. Through the graph convolution local feature fusion model, N patches are locally spatially displaced and their positions are recorded in groups. The central patch of each group is used as a node to construct a spatial similarity graph. The connection relationship of each patch is calculated in a graph convolution manner to better integrate the local features and global features of each target. The graph convolutional local feature fusion model is: The local features of the target’s N patches are further divided according to the step size s, and the division results are regarded as graph nodes. The distance between each node is expressed as: in represents the strength of the association between nodes i and j, v i represents the feature of node i, F Cosine represents the spatial cosine distance, expressed as Use spatial position and its own features to build adjacency matrix and degree matrix to complete the construction of graph neural network; S3: Mapping the targets at the edges of adjacent fields of view into overlapping areas through the overlapping area coordinate mapping model, and mapping the targets in the previous field of view to the next field of view; S4: A neighborhood trajectory priority matching model based on target movement characteristics divides the target trajectory into regions according to the spatiotemporal constraints, pre-filters the trajectories that do not meet the current target matching object, and prioritizes the trajectories that meet the requirements in the trajectory matching stage; S5: A multi-level data association model based on multi-source information fusion uses the characteristics of trajectory and target time flow data, and uses the target's appearance features, local spatial position, and global spatial position to build a three-level matching algorithm to improve the success rate of data association; S6: After the data association is completed, the data of each successfully matched trajectory is updated to obtain the information of the target at the current moment, and the target and trajectory at the next moment are matched based on this information; S7: Deploy the model in practice.

2. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 1 is characterized in that: Step S1 specifically includes the following steps: S11: Before radar-vision fusion, perform spatiotemporal synchronization of radar and vision sensors, that is, perform joint calibration in space and time alignment in time; select the world coordinate system as a bridge to jointly calibrate the radar coordinate system and the camera coordinate system; obtain the rotation angles of the three axes from the radar's external parameter matrix, set the corresponding α, β, and γ as the rotation angles of the Z axis, Y axis, and X axis, multiply each component by the corresponding rotation angle, and add the corresponding translation matrix to complete the conversion between the radar and the world coordinate system. The final conversion formula is as follows: The camera coordinate system also needs to establish a conversion relationship with the world coordinate system; the camera's extrinsic parameter matrix Rc is obtained through the camera extrinsic parameter calibration process, and the conversion formula between the camera and the world coordinate system is as follows: Then convert the camera coordinate system to the pixel coordinate system. Finally, the conversion relationship between the radar coordinate system and the pixel coordinate system is expressed as: Where X r , Y r , Z r is the radar coordinate, R is the transfer matrix between the radar coordinate system and the camera coordinate system, f is the focal length of the camera, dx and dy represent the actual length of a single pixel of the camera on the imaging sensor in the horizontal and vertical directions, and u0 and v0 represent the center position of the image; S12: Time-align the two sensors, use the timestamp of the point cloud data collected by the radar as a reference, and read the corresponding image data; S13: The data set is enhanced by a multi-target geometric fusion data enhancement method. The complete target is randomly selected to overlap other targets in geometric areas, the target itself is erased using geometric figures, the erased parts are filled with parts of other targets and marked as occluded targets to simulate the situation where the target is blocked by a large area. Small targets, incomplete targets at the edge of the field of view, small targets and occluded targets are added to the data set. The processed data set is used to train the radar and vision fusion target detection model.

3. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 1, characterized in that: The graph convolutional local feature fusion model is trained by graph convolution. The adjacency matrix and degree matrix are updated through continuous iterations to improve the connection and distinction between the local features of the target, and to improve the discrimination of the local feature differences of the target by the graph embedded domain adaptive target re-identification model. The update rules between each node are expressed as: Among them U i is represented as node features, L is represented as the position relationship matrix between nodes, D is represented as the degree matrix, A is represented as the adjacency matrix, X i Represented as the i-th local appearance feature, Represents a learnable parameter matrix used to learn local features x i The weight of the feature in the total features; The loss function uses the Triplet loss triplet loss function to calculate the inter-class difference and intra-class similarity loss, and the cross entropy loss is used as the classification loss; three targets are selected as a group to calculate the loss, A1, A2, and A3 form a group of samples, where A1 and A2 are the same target, simulating the intra-class difference; A3 and A1, A2 belong to different targets, simulating the inter-class similarity; the loss function corresponding to the triplet is as follows: Where f(x) is the target feature extraction network, are the three targets in the triplet, α is a custom distance parameter, which indicates the minimum distance between positive and negative sample pairs.

4. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 3 is characterized in that: The training process of the graph embedded domain adaptive object re-identification model is as follows: S101. Determine that the training epoch number of the deep learning model is 80; determine the learning rate of the model training, the initial learning rate is set at 0.01, and use the warmup strategy. The learning rate increases in the first 10 epochs, and the learning rate in subsequent rounds gradually decays according to the cosine decay ratio; S102. Select SGD optimizer as the model training optimizer, and select triple loss function as the loss function of the model; S103. Determine that the dimension of the target feature vector finally output by the model is 756; S104. The initial values ​​of the position code and the camera bias code are encoded using all zeros; S105. The stride of the last convolutional layer of the Patch Embedding module is 16*16; S106. Set Batch Size to 32; S107. Start training.

5. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 4 is characterized in that: The image embedded domain adaptive target re-identification model is trained in two stages through the adaptive field of view scene two-stage training strategy. During the adaptive domain training process, the re-identification model trained in the first stage is used to extract its features g for each target. I , then extract all the target features at the current moment captured by the target lens and take the feature average The characteristic average value Acting on the target feature g I Then extract the features of the target in all shots. I , and find the mean, and set the feature t I , feature g I and Features According to different weights, the final target features are formed by fusion; Based on this feature, the original training model is adjusted, and the core formula of the second stage training of the model is as follows: The eigenvalue f I That is, the target appearance feature that integrates the camera field of view feature. The final fusion feature is a fusion of the features of each angle of the target and each camera field of view; The two-stage training process is: S201. Collect pictures in the field deployment scene, build a small data set, and use the target detection model to perform target capture; S202. Grouping the captured images according to the target; S203. Grouping the captured images by camera; S204. Set Epoch to 5, the initial learning rate to 0.001, and adopt the learning rate decay strategy in each round; S205. Start training; S206. Extract features of all targets under camera i using the model trained in the first stage, and calculate the average value to obtain the feature vector S207. Traverse each single target image to obtain the target appearance feature vector g I , according to the formula Under the action of weight α, we can get f I ; S208. Traverse all images of each target under all lenses to obtain the feature vector t I , according to the formula The final target appearance is obtained under the action of weight β S209. I and Perform loss calculation, perform backpropagation, and fine-tune the model.

6. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 1, characterized in that: Step S3 specifically includes: finding four or more groups of points at the same position in the overlapping area in advance, calculating the homography matrix H representing the mapping relationship between the two fields of view, and using the homography matrix to construct the mapping relationship between the overlapping areas of the two fields of view; the target coordinate transformation relationship in the two fields of view is shown in the following formula: is the coordinate of the target center point in the overlapping area of ​​the previous field of view. After the transformation of the coordinate mapping relationship matrix H, the coordinate in the next field of view is obtained.

7. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 1, characterized in that: Step S4 specifically includes the following steps: S41: First, use the movement direction of the target detected by the radar to filter the trajectory when matching the trajectory. When it is determined that there is no turn, the target with the same movement direction is matched first. S42: filtering according to the moving areas adjacent to the target, filtering the trajectory in the area that the target cannot reach in a short time; for each camera coverage area, demarcating the area according to traffic regulations, and indicating the area where the target can move continuously.

8. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 1, characterized in that: In step S5, before matching the target with the trajectory, Kalman filtering is used to predict the position of the trajectory at the previous moment at the current moment, and the position information is assigned to the trajectory; two different motion models are established for the target motion state, namely, a constant rate motion model and a uniform linear motion model, and the model parameters are the lateral position, longitudinal position and their respective first-order derivatives of the target. The other model is a constant turn rate and speed model, and the model parameters include the target's lateral and longitudinal positions and steering angles as well as their respective first-order derivatives. The two models are adaptively selected by the interactive multi-model Kalman filter at different stages according to the target's motion state to predict and update the target's motion state.

9. The method for tracking multiple targets across multiple domains at intersections based on overlapping fields of view according to claim 1, characterized in that: Step S6 specifically includes: firstly, using the Mahalanobis space distance between the trajectory and the target appearance feature vector to construct a cost matrix, and performing the first-level matching between the target and the trajectory. For the target and trajectory that failed to match, directly enter the second-level matching. The target and trajectory that successfully matched further determine whether their field of view IDs are the same. If they are the same, it indicates that the target has not crossed the field of view, and enter the second-level matching for auxiliary matching. If the field of view IDs are different, it indicates that the target has crossed the domain. At this time, the cost matrix is ​​calculated using the IOU of the target's mapping bounding box and the bounding box of the trajectory for matching; If both the first and second level matches are successful, the target is deemed to be matched with the track successfully, and data association can be performed to update the track with the target information at the current moment; If the match fails, the third level matching is entered, and the state vector is constructed using the absolute position information of the latitude and longitude of the target and the trajectory, the target speed, and the driving angle to calculate the cost matrix for matching; The data association module of the target and trajectory is constructed through three-level matching.

Images