Cross-view multi-target tracking method, device, electronic device and storage medium

Abstract

The present disclosure provides a cross-view multi-target tracking method, device, electronic device and storage medium, which includes: at each moment in the tracking process, executing the following steps: performing target detection processing on the current frames of videos of different viewpoints respectively to obtain target information of each target of each viewpoint; using a single-view feature extraction network to extract single-view features from the target information; using a cross-view feature extraction network to extract cross-view features from the target information; constructing an undirected graph, wherein the nodes in the undirected graph include the target nodes of each target of each viewpoint detected at the current moment, and the trajectory nodes of each target trajectory currently determined; according to the attributes of each node, dividing the nodes corresponding to the same target in the undirected graph into the same subgraph; based on the attributes of each node in each subgraph, determining the target trajectory corresponding to the subgraph and the attributes of the target trajectory.

Description

Technical Field

[0001] The present disclosure generally relates to the field of target tracking technology, and more specifically, to a method, device, electronic device, and storage medium for cross-view multi-target tracking. Background Art

[0002] In the fields of computer vision and artificial intelligence, cross-view multi-object tracking (CMT) is a key technology for numerous applications, including intelligent surveillance, traffic management, and robotic navigation. Taking urban traffic monitoring as an example, real-time tracking of vehicles and pedestrians using multi-view cameras effectively addresses issues such as object occlusion and view limitations in a single viewpoint, improving the accuracy and reliability of monitoring systems and providing strong support for intelligent traffic management. However, CMT presents numerous challenges, particularly in feature extraction, object association, and trajectory generation.

[0003] Feature extraction: Feature extraction is a key step in cross-view multi-target tracking. Because target appearance and geometric relationships vary significantly across different viewpoints, traditional feature extraction methods struggle to adapt to these variations. For example, deep learning-based feature extraction networks are often affected by viewpoint variations, resulting in large feature distance gaps and difficulty in setting a universal association threshold. Furthermore, existing methods often rely on stable camera viewpoints and homography matrices from specific datasets, limiting their applicability in open scenarios.

[0004] Target association: Target association is a core task in cross-view multi-target tracking. Existing methods typically employ appearance-based matching or geometric constraints to accomplish this task. However, in multi-view scenarios, the order of appearance, appearance features, and motion models of targets vary depending on the viewpoint, making target association more complex. For example, traditional matching methods based on the Hungarian algorithm struggle to handle the spatiotemporal constraints and geometric relationships involved in cross-view target association, resulting in reduced association accuracy.

[0005] Trajectory Generation: Trajectory generation is the ultimate goal of cross-view multi-target tracking. Existing methods typically treat target association and trajectory generation as independent steps, ignoring the interaction between cross-view information and single-view information. For example, while multi-target tracking methods based on graph models can leverage graph structures for optimization, effectively fusing multi-view information to generate stable trajectories in cross-view scenarios remains an unsolved problem. Furthermore, existing methods often lack the use of global information during trajectory generation, resulting in insufficient trajectory continuity and stability.

[0006] In summary, existing cross-view multi-target tracking technologies still have many problems in feature extraction, target association, and trajectory generation, making it difficult to meet the high-precision, high-robustness, and real-time requirements of practical applications. Therefore, developing a cross-view multi-target tracking method that can effectively address these issues has important research significance and application value. Summary of the Invention

[0007] The present disclosure provides a cross-view multi-target tracking method, device, electronic device and storage medium to solve at least one of the above problems.

[0008] According to a first aspect of an embodiment of the present disclosure, a cross-view multi-target tracking method is provided, comprising: at each moment in the tracking process, executing the following steps: performing target detection processing on current frames of videos of different viewpoints respectively to obtain target information of each target of each viewpoint; extracting single-view features from the target information using a single-view feature extraction network; extracting cross-view features from the target information using a cross-view feature extraction network, wherein the cross-view feature extraction network is used to extract similar features for the same target of different viewpoints; constructing an undirected graph, wherein the nodes in the undirected graph include target nodes of each target of each viewpoint detected at the current moment, and trajectory nodes of each target trajectory currently determined, wherein the target trajectory is the motion path of the same target in a time series; segmenting the nodes in the undirected graph corresponding to the same target into the same subgraph according to the attributes of each node, wherein the attributes include cross-view features and single-view features of each viewpoint involved in the corresponding node; and determining the target trajectory corresponding to the subgraph and the attributes of the target trajectory based on the attributes of each node in each subgraph.

[0009] Optionally, the undirected graph also includes an edge for connecting two nodes, wherein the nodes in the undirected graph corresponding to the same target are divided into the same subgraph according to the attributes of each node, including: for each edge in the undirected graph, determining the edge weight of the edge according to the attributes of the two nodes connected by the edge, wherein the edge weight is used to represent the possibility that the corresponding two nodes correspond to the same target; based on the edge weight of each edge in the undirected graph, the undirected graph is divided into multiple non-intersecting subgraphs by solving the minimum multicut problem of the undirected graph.

[0010] Optionally, the attributes also include location information, wherein determining the edge weight of the edge based on the attributes of the two nodes connected by the edge includes: determining the position distance of the two nodes based on the position information of the two nodes connected by the edge as the position weight of the edge; determining the feature similarity of the two nodes based on at least one of the single-perspective features and cross-perspective features of the two nodes connected by the edge as the feature weight of the edge; and determining the edge weight of the edge based on the position weight and feature weight of the edge.

[0011] Optionally, determining the edge weight of the edge based on the position weight and feature weight of the edge includes: when the position distance between the two nodes is less than a distance threshold, determining the weighted sum of the position weight and the feature weight of the edge as the edge weight of the edge; when the position distance between the two nodes is greater than or equal to the distance threshold, determining the difference between the weighted sum and a preset penalty value as the edge weight of the edge.

[0012] Optionally, the method of determining the degree of feature similarity between the two nodes connected by the edge as the feature weight of the edge based on at least one of the single-perspective features and cross-perspective features of the two nodes includes: when the two nodes connected by the edge are target nodes of different perspectives, determining the degree of feature similarity between the two nodes as the feature weight of the edge based on the cross-perspective features of the two nodes; when the two nodes connected by the edge are a target node and a trajectory node respectively, taking the perspective to which the target node belongs as a reference perspective; for the reference perspective, determining the degree of feature similarity between the two nodes in the reference perspective based on the single-perspective features of the two nodes; for each perspective other than the reference perspective, determining the degree of feature similarity between the two nodes in each other perspective based on the cross-perspective features of the two nodes; and determining the feature weight of the edge based on the degree of feature similarity in each perspective.

[0013] Optionally, the single-view feature extraction network is trained by the following steps: obtaining image frames of a video of the same view at two preceding and following moments, recording the image frame at the preceding moment as a preceding sample frame, and recording the image frame at the following moment as a following sample frame, wherein the preceding sample frame carries a true target label; performing target detection processing on the preceding sample frame to obtain target information of the target corresponding to the true target label as the preceding sample target information; using the single-view feature extraction network to be trained to extract the preceding sample single-view features from the preceding sample target information; performing target detection processing on the following sample frame The invention relates to a method for extracting single-view features of the single-view feature extraction network to be trained, and a method for extracting single-view features of the single-view feature extraction network to be trained. The method comprises the following steps: performing target detection processing to obtain target information of multiple candidate frames as subsequent sample target information of each candidate frame; using the single-view feature extraction network to be trained to extract subsequent sample single-view features from the subsequent sample target information of each candidate frame; performing cross-correlation calculation on the single-view features of the prior sample and the single-view features of the subsequent sample of each candidate frame to obtain a time clue matrix; determining a loss value according to the time clue matrix and the true target label; and adjusting the parameters of the single-view feature extraction network to be trained according to the loss value to obtain the single-view feature extraction network.

[0014] According to a second aspect of an embodiment of the present disclosure, a cross-view multi-target tracking apparatus is provided, comprising: at each moment in a tracking process, calling the following units: a detection unit, configured to perform target detection processing on current frames of videos of different viewpoints, respectively, to obtain target information of each target of each viewpoint; a first extraction unit, configured to use a single-view feature extraction network to extract single-view features from the target information; a second extraction unit, configured to use a cross-view feature extraction network to extract cross-view features from the target information, wherein the cross-view feature extraction network is used to extract similar features for the same target of different viewpoints; and a construction unit, It is configured to construct an undirected graph, wherein the nodes in the undirected graph include a target node for each target of each perspective detected at the current moment, and a trajectory node for each target trajectory currently determined, wherein the target trajectory is the motion path of the same target in the time series; the segmentation unit is configured to segment the nodes corresponding to the same target in the undirected graph into the same subgraph according to the attributes of each node, wherein the attributes include cross-perspective features and single-perspective features of each perspective involved in the corresponding node; the determination unit is configured to determine the target trajectory corresponding to the subgraph and the attributes of the target trajectory based on the attributes of each node in each subgraph.

[0015] Optionally, the undirected graph also includes an edge for connecting two nodes, and the segmentation unit is further configured to: for each edge in the undirected graph, determine the edge weight of the edge according to the attributes of the two nodes connected by the edge, wherein the edge weight is used to represent the possibility that the corresponding two nodes correspond to the same target; based on the edge weight of each edge in the undirected graph, the undirected graph is segmented into multiple non-intersecting subgraphs by solving the minimum multicut problem of the undirected graph.

[0016] Optionally, the attributes also include location information, and the segmentation unit is further configured to: determine the location distance of the two nodes connected by the edge based on the location information of the two nodes as the location weight of the edge; determine the feature similarity of the two nodes based on at least one of the single-perspective features and cross-perspective features of the two nodes connected by the edge as the feature weight of the edge; determine the edge weight of the edge based on the location weight and feature weight of the edge.

[0017] Optionally, the segmentation unit is further configured to: when the position distance between the two nodes is less than a distance threshold, determine the weighted sum of the position weight and the feature weight of the edge as the edge weight of the edge; when the position distance between the two nodes is greater than or equal to the distance threshold, determine the difference between the weighted sum and a preset penalty value as the edge weight of the edge.

[0018] Optionally, the segmentation unit is further configured to: when the two nodes connected by the edge are target nodes of different perspectives, determine the feature similarity of the two nodes according to the cross-perspective features of the two nodes as the feature weight of the edge; when the two nodes connected by the edge are respectively a target node and a trajectory node, take the perspective to which the target node belongs as the reference perspective; for the reference perspective, determine the feature similarity of the two nodes in the reference perspective according to the single-perspective features of the two nodes; for each perspective other than the reference perspective, determine the feature similarity of the two nodes in each other perspective according to the cross-perspective features of the two nodes; and determine the feature weight of the edge according to the feature similarity of each perspective.

[0019] Optionally, the single-view feature extraction network is trained by the following steps: obtaining image frames of a video of the same view at two preceding and following moments, recording the image frame at the preceding moment as a preceding sample frame, and recording the image frame at the following moment as a following sample frame, wherein the preceding sample frame carries a true target label; performing target detection processing on the preceding sample frame to obtain target information of the target corresponding to the true target label as the preceding sample target information; using the single-view feature extraction network to be trained to extract the preceding sample single-view features from the preceding sample target information; performing target detection processing on the following sample frame The invention relates to a method for extracting single-view features of the single-view feature extraction network to be trained, and a method for extracting single-view features of the single-view feature extraction network to be trained. The method comprises the following steps: performing target detection processing to obtain target information of multiple candidate frames as subsequent sample target information of each candidate frame; using the single-view feature extraction network to be trained to extract subsequent sample single-view features from the subsequent sample target information of each candidate frame; performing cross-correlation calculation on the single-view features of the prior sample and the single-view features of the subsequent sample of each candidate frame to obtain a time clue matrix; determining a loss value according to the time clue matrix and the true target label; and adjusting the parameters of the single-view feature extraction network to be trained according to the loss value to obtain the single-view feature extraction network.

[0020] According to a third aspect of an embodiment of the present disclosure, an electronic device is provided, comprising: at least one processor; and at least one memory storing computer-executable instructions, wherein the computer-executable instructions, when executed by the at least one processor, prompt the at least one processor to execute a cross-view multi-target tracking method according to an exemplary embodiment of the present disclosure.

[0021] According to a fourth aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided. When instructions in the computer-readable storage medium are executed by at least one processor, the at least one processor is prompted to execute a cross-view multi-target tracking method according to an exemplary embodiment of the present disclosure.

[0022] According to a fifth aspect of an embodiment of the present disclosure, a computer program product is provided, comprising computer instructions, which, when executed by at least one processor, prompt the at least one processor to execute a cross-view multi-target tracking method according to an exemplary embodiment of the present disclosure.

[0023] The technical solutions provided by the embodiments of the present disclosure provide at least the following beneficial effects: The cross-view multi-target tracking method, device, electronic device, and storage medium disclosed herein improve tracking accuracy and robustness by optimizing feature extraction and target association strategies, while reducing network complexity and accelerating processing speed. Specifically, by extracting single-view and cross-view features of a target in parallel, the feature distance intervals for the same target under different viewpoints can be reduced, while also extracting richer features and improving feature expression capabilities. Furthermore, by utilizing a single-view feature extraction network and a cross-view feature extraction network to perform their respective feature extractions, the number of network layers and parameters is reduced, helping to reduce network complexity and speeding up network training and testing. Furthermore, by constructing nodes of an undirected graph based on the currently detected target and the currently determined target trajectory, and then partitioning the undirected graph according to the node attributes, and clustering these nodes, the corresponding relationships between each target under different viewpoints and the corresponding relationships between the target and the target trajectory can be simultaneously determined, achieving view alignment and motion alignment, thereby fully utilizing the interaction between cross-view and single-view features to simultaneously achieve target association and trajectory generation. At the same time, the target trajectory is gradually expanded over time, which can optimize target association and trajectory generation, and improve tracking accuracy and robustness.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The accompanying drawings herein are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present disclosure, and together with the description are used to explain the principles of the present disclosure, and do not constitute an improper limitation of the present disclosure.

[0026] Figure 1 4 is a flowchart of a cross-view multi-target tracking method according to an exemplary embodiment of the present disclosure.

[0027] Figure 2 2 is a schematic structural diagram of a single-view feature extraction network or a cross-view feature extraction network according to an exemplary embodiment of the present disclosure.

[0028] Figure 3 is a schematic diagram of partitioning an undirected graph according to an exemplary embodiment of the present disclosure.

[0029] Figure 4 It is a schematic flowchart of a cross-view multi-target tracking method according to a specific embodiment of the present disclosure.

[0030] Figure 5 4 is a block diagram of a cross-view multi-target tracking apparatus according to an exemplary embodiment of the present disclosure.

[0031] Figure 6 is a block diagram of an electronic device according to an exemplary embodiment of the present disclosure. DETAILED DESCRIPTION

[0032] In order to enable ordinary persons in the art to better understand the technical solutions of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below with reference to the accompanying drawings.

[0033] It should be noted that the terms "first," "second," and the like in the specification and claims of the present disclosure and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the numbers used in this manner are interchangeable where appropriate so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. The implementation methods described in the following examples do not represent all implementation methods consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with certain aspects of the present disclosure as detailed in the appended claims.

[0034] It should be noted that the phrase "at least one of the several items" in this disclosure includes three types of parallel situations: "any one of the several items", "a combination of any multiple of the several items", and "all of the several items". For example, "including at least one of A and B" includes the following three parallel situations: (1) including A; (2) including B; (3) including A and B. For another example, "performing at least one of step 1 and step 2" means the following three parallel situations: (1) performing step 1; (2) performing step 2; and (3) performing both step 1 and step 2.

[0035] Hereinafter, a method, apparatus, electronic device, and storage medium for tracking multiple targets across perspectives according to exemplary embodiments of the present disclosure will be described in detail with reference to the accompanying drawings.

[0036] Figure 1 is a flow chart of a cross-view multi-target tracking method according to an exemplary embodiment of the present disclosure. The method can be executed on an electronic device with sufficient computing power and refers to Figure 1 The entire process can be executed cyclically during the tracking process, specifically, the entire process is executed once at each execution moment, and the target trajectory information obtained by the previous execution can be continuously accumulated for use at subsequent moments. It should be understood that Figure 1 If the drawing is not completed, the tracking can be ended in response to the end instruction.

[0037] Reference Figure 1 In step S101, target detection processing is performed on the current frames of videos of different perspectives respectively to obtain target information of each target in each perspective.

[0038] As an example, OpenCV's videoCapture function can be used to extract image frame information from a video, thereby obtaining the current frame for each perspective. Object detection is then performed on each obtained image frame, for example using the open-source YOLOX-X (You Only Look Once eXtreme) object detection algorithm. This method generates multi-scale high-dimensional features with dimensions of H × W × C, where H and W represent the height and width of the image frame, respectively, and C represents the feature size of the high-dimensional features, for example, 512. The obtained high-dimensional features are passed through the head network of the object detection model to obtain the object's location information (x, y, w, h), classification prediction, and confidence prediction. The target information obtained in step S101 can specifically be this high-dimensional feature, that is, the intermediate feature obtained during the object detection process.

[0039] In step S102, a single-view feature extraction network is used to extract single-view features from target information.

[0040] This step models the target motion features under a single view through a single view feature extraction network.

[0041] Optionally, the single-view feature extraction network is trained by the following steps: obtaining image frames of a video of the same view at two preceding and succeeding moments, recording the image frame at the preceding moment as the preceding sample frame, and recording the image frame at the succeeding moment as the succeeding sample frame, wherein the preceding sample frame carries a true target label; performing target detection processing on the preceding sample frame to obtain target information of the target corresponding to the true target label as the preceding sample target information; using the single-view feature extraction network to be trained, extracting preceding sample single-view features from the preceding sample target information; performing target detection processing on the succeeding sample frame to obtain target information of multiple candidate frames as the succeeding sample target information of each candidate frame; using the single-view feature extraction network to be trained, extracting succeeding sample single-view features from the succeeding sample target information of each candidate frame; performing cross-correlation calculation on the preceding sample single-view features and the succeeding sample single-view features of each candidate frame to obtain a temporal cue matrix; determining a loss value based on the temporal cue matrix and the true target label; adjusting the parameters of the single-view feature extraction network to be trained based on the loss value to obtain a single-view feature extraction network. By calculating the temporal cue matrix, the features of the image frames at the previous and next moments can be fused, and the single-view feature extraction network can be trained based on this, which enables the network to learn how to capture the dynamic information of the target, facilitating the effective association between the target and the target trajectory in target tracking.

[0042] As an example, the structure of the single-view feature extraction network is as follows Figure 2As shown, a deep learning network structure based on a 2D convolution (Conv2d)-normalization (Batch Normalization, abbreviated as BN)-activation function (such as a rectified linear unit, abbreviated as ReLU) structure can be adopted, and specifically can include two of the deep learning network structures and a 2D convolution structure.

[0043] As an example, during the training process, target detection processing can be performed on the prior sample frame, and several candidate frames and their corresponding high-dimensional features can be obtained through the detector. Based on the candidate frame position of the real target label carried by the prior sample frame, the detected candidate frame corresponding to the real target and its corresponding high-dimensional features of size 1×C are obtained as the prior sample target information. After passing through the single-view feature extraction network to be trained, the prior sample single-view features of size 1×C' can be obtained. For the subsequent sample frame, the anchor-free detector can be used to output a high-dimensional feature matrix of size H×W×C as the subsequent sample target information of the multiple candidate frames as a whole. Each element in the matrix represents a candidate frame, and the feature size is C. After the high-dimensional feature matrix is ​​processed by the single-view feature extraction network, a subsequent sample single-view feature matrix of size H×W×C' can be obtained. Since the dimension of the single-view feature of the prior sample is 1×C', the cross-correlation calculation operation in a single target is reduced to matrix multiplication. The matrix multiplication is performed by the transpose of the single-view feature matrix of the subsequent sample and the single-view feature of the prior sample to obtain a time clue matrix of size H×W×1.

[0044] As an example, regarding the loss function in training, the Logistic-MSE loss function can be used, and the formula is as follows.

[0045]

[0046] Since the detector will obtain feature maps of multiple scales, the loss function needs to be calculated for each feature map. In the above formula, Represents the total number of candidate boxes detected in the subsequent sample frame, represents the calculated time clue matrix, represents the Gaussian function matrix generated using the true target label, Represents the coordinate point in the image frame, represents the t-th frame image, The response map representing the true target labels, Represents the coordinate point in the calculated time clue matrix Response map of the true target label Generated by Gaussian function, the formula is as follows.

[0047]

[0048] in Indicates the standard deviation of the Gaussian function, which can be selected based on experience, such as 0.75. , ) represents the true value position of the target, specifically the true center position of the target, which is recorded in the true target label. This formula indicates that the response value is 1 at the true center position of the target, and the farther away from the true center position other positions are, the lower the response value.

[0049] Return to reference Figure 1 ,In step S103, a cross-view feature extraction network is used to extract cross-view features from the target information.

[0050] The cross-view feature extraction network is used to extract similar features for the same target at different viewpoints, which can reduce the feature distance interval of the same target at different viewpoints.

[0051] Optionally, a cross-view feature extraction network is trained by the following steps: obtaining image frames of videos from different viewpoints at the same time as sample frames; performing target detection processing on each sample frame to obtain target information of each target in each sample frame as sample target information; extracting sample cross-view features from each sample target information using the cross-view feature extraction network to be trained; determining the value of a cross-view consistency loss function based on the extracted sample cross-view features as a loss value; and adjusting the parameters of the cross-view feature extraction network to be trained based on the loss value to obtain a cross-view feature extraction network. By introducing a cross-view consistency loss function to train the cross-view feature extraction network, the similarity of features of the same target under different viewpoints can be enhanced, while suppressing the similarity of features between different targets, thereby reducing the distance interval between cross-view features extracted by the network under different viewpoints.

[0052] As an example, the structure of the cross-view feature extraction network can also be as follows Figure 2 As shown, no further details are given here.

[0053] As an example, the formula of the cross-view consistency loss function is as follows.

[0054]

[0055] In the above formula, N and M are the image frames of view 1 (hereinafter referred to as view Figure 1 ) and image frames of view 2 (hereinafter referred to as view Figure 2 ) is the number of detected objects in . is an indicator function, only when Figure 1 The i-th target and view Figure 2The value of is 1 if the jth target in represents the same object, and 0 otherwise. Respectively represent the Figure 1 The cross-view features of the i-th target, Figure 2 The cross-view features of the jth object. D represents the distance function, which is the cosine distance here. It should be noted that the cross-view consistency loss function is calculated based on view pairs (i.e., two views). If the number of view pairs is greater than two, multiple view pairs can be determined, and the cross-view consistency loss can be calculated for each view pair. The multiple cross-view consistency losses are then combined, for example, including but not limited to calculating a sum or weighted sum, to form the final cross-view consistency loss.

[0056] The cross-view multi-target tracking method of the exemplary embodiment of the present disclosure extracts single-view and cross-view features of the target in parallel, reducing the distance between features of the same target under different viewpoints while also extracting richer features and improving feature expression capabilities. Furthermore, by utilizing a single-view feature extraction network and a cross-view feature extraction network to extract features separately, the number of network layers and parameters is reduced, helping to reduce network complexity and speeding up network training and testing.

[0057] Return to reference Figure 1 , in step S104, an undirected graph is constructed.

[0058] The nodes in the undirected graph include a target node for each target detected at each viewpoint at the current moment, and a track node for each currently determined target track. A target node represents an independent target detected at the current moment, while a target track represents the motion path of the same target over time. It should be noted that at the initial moment, no target tracks have been determined, so the constructed undirected graph only contains target nodes, not track nodes. At the second moment, the target detected at the initial moment can be considered a target track.

[0059] In step S105 , nodes corresponding to the same target in the undirected graph are divided into the same subgraph according to the attributes of each node.

[0060] Attributes include cross-view features and single-view features for each viewpoint involved in the corresponding node. Specifically, since the target node has a specific viewpoint, it only involves cross-view features and single-view features for that viewpoint. Trajectory nodes, on the other hand, involve multiple different viewpoints, as the target trajectory is derived from targets at different viewpoints, and each viewpoint contains cross-view features and single-view features.

[0061] When segmenting an undirected graph, each target node corresponds to a clear target, and each target trajectory also corresponds to a clear target. Therefore, the nodes in the undirected graph can be segmented by predicting whether two nodes correspond to the same target.

[0062] In step S106 , based on the attributes of each node in each subgraph, the target trajectory corresponding to the subgraph and the attributes of the target trajectory are determined.

[0063] Specifically, each subgraph contains at most one trajectory node and at most one target node from each perspective. For each subgraph, if the subgraph only contains a trajectory node, it indicates that the target trajectory has not found a target in the target association at the current moment. Therefore, its trajectory status can be set to lost and the target trajectory can be discarded after a certain loss duration. If the subgraph only contains a target node, it indicates that the target is a newly appeared target. The confidence level predicted during the target detection process can be used to determine whether the target can be used as a new target trajectory. In other words, if the confidence level of the target is sufficiently high, for example, greater than a preset confidence threshold, the target is used as a new target trajectory, and the attributes of the target from each perspective are used as the attributes of the new target trajectory. Otherwise, the target is discarded. If the subgraph contains both target nodes and trajectory nodes, it indicates that the tracked target trajectory can be updated. The target node information is added to the trajectory node to serve as the updated target trajectory. In the case of updating the target trajectory, the properties of the target trajectory also need to be updated. At this time, for each perspective, the cross-perspective features of the target node in that perspective can be fused with the cross-perspective features of the trajectory node in that perspective, and the single-perspective features of the target node in that perspective can be fused with the single-perspective features of the trajectory node in that perspective, for example, using a linear fusion method, which is not limited in this disclosure. Figure 3 A specific example of partitioning an undirected graph is shown. Figure 3 In , the hollow circles represent target nodes, and different line types (i.e., thick solid lines and thin solid lines) are used to distinguish different perspectives. That is, hollow circles of the same line type represent targets detected under the same perspective, e.g. Figure 3 The three target nodes obj1, obj2, and obj3 under the same view are marked in the figure; the solid circles represent the trajectory nodes, which are marked as Tra1, Tra2, Tra3, and Tra4 respectively. Among them, the trajectory node Tra4 (in order to distinguish it from other trajectory nodes, Figure 3 Tra4 is colored black in the figure. After the algorithm is executed, it is not connected to other nodes, indicating that there is no target associated with it in this frame. The dotted edge connects two trajectory nodes. Based on the previous introduction to edge weights, the edge weights between trajectory nodes will introduce a preset penalty value, resulting in its edge weight being negative infinity. Therefore, this edge will be deleted after the algorithm is executed, so it is represented by a dotted line before the algorithm is executed.

[0064] According to the exemplary embodiment of the present disclosure, the cross-view multi-target tracking method constructs nodes of an undirected graph based on the currently detected targets and the currently determined target trajectories, and then segments the undirected graph according to the node attributes to cluster these nodes. This method can simultaneously determine the correspondence between each target under different viewpoints, as well as the correspondence between targets and target trajectories, achieving viewpoint alignment and motion alignment. This method fully utilizes the interaction between cross-view features and single-view features, and simultaneously achieves target association and trajectory generation. Furthermore, the target trajectories are gradually expanded over time, which can optimize target association and trajectory generation, improving tracking accuracy and robustness.

[0065] It should be noted that the order of steps in the various processes introduced in this disclosure is only for the purpose of distinguishing different steps, and is not intended to limit the order in which the steps must be executed. The order in which the steps must be executed can be adjusted if the logic is reasonable. Figure 1 Step S102 and step S103 may be executed sequentially or in parallel in any order, and the present disclosure does not impose any limitation on this.

[0066] Next, step S105 will be further introduced.

[0067] Optionally, the undirected graph also includes an edge for connecting two nodes, and step S105 includes: for each edge in the undirected graph, determining the edge weight of the edge according to the attributes of the two nodes connected by the edge, wherein the edge weight is used to represent the possibility that the corresponding two nodes correspond to the same target; based on the edge weight of each edge in the undirected graph, the undirected graph is divided into multiple non-intersecting subgraphs by solving the minimum multi-cut problem of the undirected graph. By modeling the target association task as an optimized solution of the minimum multi-cut problem in the tracking stage, a stable and accurate target trajectory can be generated, providing a more accurate and robust solution for cross-view target tracking. Specifically, the minimum multi-cut problem is to divide the undirected graph into multiple non-intersecting subgraphs by cutting the edges in the undirected graph, and the goal of cutting is to find a cutting scheme that minimizes the sum of the edge weights of the cut edges.

[0068] The minimum multi-cut problem can be expressed using the following formulas.

[0069]

[0070]

[0071]

[0072]

[0073] In the above formulas, G represents an undirected graph, V represents a node in the undirected graph, E represents an edge in the undirected graph, and w is the edge weight. This is implemented using the lifted_multicut function of the nifty library. is the edge to be deleted, The edge to be deleted The following two formulas indicate that for any circle in an undirected graph , assuming If it is 1 (meaning it needs to be deleted), then at least one more edge should be deleted from the circle in addition to this edge. This restriction is to ensure that when an edge is deleted, the nodes at both ends of the edge are no longer connected by other edges, so as to achieve the clustering effect.

[0074] Optionally, the attributes also include position information, which can be obtained based on the position information obtained during the aforementioned target detection process. To this end, when updating the attributes of the target trajectory in step S106, for example, the predicted position and the observed position under the same viewing angle can be subjected to a Kalman filter to update the position information. In this case, the operation of determining the edge weight of the edge based on the attributes of the two nodes connected by the edge in step S105 can include the following three steps.

[0075] The first step is to determine the position distance between the two nodes based on the position information of the two nodes connected by the edge, which is used as the position weight of the edge.

[0076] Specifically, position information is used to describe the spatial location of the target in the image, typically including the coordinates of the candidate box or the projected position of the target in a planar view. During the first step, the position weight can be measured by calculating the spatial proximity of two nodes.

[0077] As an example, for targets within the same perspective, the location information may include the center point coordinates and size of the candidate box; for targets across perspectives, the location information can be used to project the target into a unified planar view coordinate system through the homography matrix and calculate its center point distance.

[0078] In the second step, the feature similarity between the two nodes connected by the edge is determined based on at least one of the single-view features and cross-view features of the two nodes, which is used as the feature weight of the edge.

[0079] Specifically, single-view features and cross-view features are used to characterize the appearance characteristics of the target and are obtained by single-view feature extraction networks and cross-view feature extraction networks.

[0080] Optionally, the second step includes: when the two nodes connected by the edge are target nodes from different perspectives, determining the degree of feature similarity between the two nodes based on the cross-perspective features of the two nodes as the feature weight of the edge; when the two nodes connected by the edge are the target node and the trajectory node respectively, taking the perspective to which the target node belongs as the reference perspective; for the reference perspective, determining the degree of feature similarity between the two nodes in the reference perspective based on the single-perspective features of the two nodes; for each perspective other than the reference perspective, determining the degree of feature similarity between the two nodes in each other perspective based on the cross-perspective features of the two nodes; and determining the feature weight of the edge based on the degree of feature similarity for each perspective. By reasonably selecting single-perspective features or multi-perspective features for different node types and specific perspectives to calculate the similarity between the feature vectors of the two nodes, and measuring the feature weight accordingly, reliable feature weights can be obtained. As an example, cosine similarity can be used as a measure of feature similarity.

[0081] The third step is to determine the edge weight based on the edge's position weight and feature weight.

[0082] By combining position weights and feature weights to determine edge weights, the calculation of edge weights can comprehensively consider the spatial position and appearance characteristics of the target, thereby comprehensively evaluating the similarity and correlation between nodes, which helps to optimize target association and trajectory generation.

[0083] Optionally, the third step includes: when the position distance between the two nodes is less than a distance threshold, determining a weighted sum of the position weight and the feature weight of the edge as the edge weight of the edge; when the position distance between the two nodes is greater than or equal to the distance threshold, determining the difference between the weighted sum and a preset penalty value as the edge weight of the edge. In the case where the position distance between the two nodes is large, the two nodes are highly likely to correspond to different targets. By further subtracting the preset penalty value to correct the edge weight in this case, the edge between the two nodes can be preferentially cut when solving the minimum multi-cut problem, thereby improving the segmentation quality and efficiency.

[0084] As an example, the edge weight is calculated by weighted summing the feature weight and the position weight as follows.

[0085]

[0086] In the above formula, Represents the edge weight, i and j represent the two nodes connected by the calculated edge, The weights used to control the influence of feature weights and position weights are shown in the following example. The feat function uses cosine distance to calculate the distance between the feature vectors of two nodes. The IoU function is used to calculate the position weights. The penalty is a preset penalty value, set when the distance between the estimated position center point on the ground plane exceeds the distance threshold. The distance threshold can be set to the square root of the area of ​​the candidate box projected onto the ground perspective multiplied by 2. This is a rough empirical value, but it can also be set to other values. If the distance threshold is exceeded, the two candidate boxes are considered to point to different targets, and the penalty can be set to positive infinity.

[0087] Figure 4 It is a schematic flowchart of a cross-view multi-target tracking method according to a specific embodiment of the present disclosure.

[0088] In this embodiment, the method is implemented using any programming language, regardless of the hardware or programming language used. This embodiment utilizes a 3.2GHz CPU, 16GB of RAM, and at least one GPU with 24GB of video memory, using Python 3.7 and the PyTorch 1.9.0 deep learning framework to implement the method.

[0089] Reference Figure 4 ,The cross-view multi-target tracking method of this specific embodiment mainly includes 4 steps.

[0090] Step 1: Extract the feature information and location information of the target from the current frame of the multi-view video.

[0091] Step 2: Model the single-view features through the single-view feature extraction network and capture the dynamic information of the target through the response matrix.

[0092] Step 3: Introduce cross-view consistency loss through the cross-view feature extraction network to reduce the feature distance interval under different viewpoints and improve the multi-view object association performance.

[0093] Step 4: Convert the target association phase into a minimum multi-cut problem for an undirected graph and model the tracking task as a global optimization problem.

[0094] Figure 5 is a block diagram of a cross-view multi-target tracking apparatus according to an exemplary embodiment of the present disclosure. Figure 5 The cross-view multi-target tracking device 500 includes a detection unit 501, a first extraction unit 502, a second extraction unit 503, a construction unit 504, a segmentation unit 505, and a determination unit 506. Figure 1 The various steps in the tracking process can be executed cyclically. Similarly, the various units in the cross-view multi-target tracking device 500 can also be called cyclically during the tracking process.

[0095] The detection unit 501 may perform target detection processing on current frames of videos of different perspectives respectively, and obtain target information of each target of each perspective.

[0096] The first extraction unit 502 may use a single-view feature extraction network to extract single-view features from the target information.

[0097] The second extraction unit 503 may use a cross-view feature extraction network to extract cross-view features from the target information, wherein the cross-view feature extraction network is used to extract similar features for the same target at different views.

[0098] The construction unit 504 can construct an undirected graph, wherein the nodes in the undirected graph include a target node for each target of each perspective detected at the current moment, and a trajectory node for each target trajectory currently determined, where the target trajectory is the motion path of the same target in a time series.

[0099] The segmentation unit 505 may segment the nodes corresponding to the same target in the undirected graph into the same subgraph according to the attributes of each node, wherein the attributes include cross-view features and single-view features of each view involved in the corresponding node.

[0100] The determining unit 506 may determine the target trajectory corresponding to the subgraph and the attributes of the target trajectory based on the attributes of each node in each subgraph.

[0101] Optionally, the undirected graph also includes an edge for connecting two nodes, and the segmentation unit 505 can also: determine the edge weight of each edge in the undirected graph according to the attributes of the two nodes connected by the edge, wherein the edge weight is used to represent the possibility that the corresponding two nodes correspond to the same target; based on the edge weight of each edge in the undirected graph, the undirected graph is segmented into multiple non-intersecting subgraphs by solving the minimum multi-cut problem of the undirected graph.

[0102] Optionally, the attributes also include position information, and the segmentation unit 505 can also: determine the position distance of the two nodes connected by the edge based on the position information of the two nodes, as the position weight of the edge; determine the feature similarity of the two nodes based on at least one of the single-view features and cross-view features of the two nodes connected by the edge, as the feature weight of the edge; determine the edge weight of the edge based on the position weight and feature weight of the edge.

[0103] Optionally, the segmentation unit 505 may also: when the position distance between the two nodes is less than a distance threshold, determine the weighted sum of the position weight and the feature weight of the edge as the edge weight of the edge; when the position distance between the two nodes is greater than or equal to the distance threshold, determine the difference between the weighted sum and the preset penalty value as the edge weight of the edge.

[0104] Optionally, the segmentation unit 505 may also: when the two nodes connected by the edge are target nodes of different perspectives, determine the feature similarity of the two nodes according to the cross-perspective features of the two nodes as the feature weight of the edge; when the two nodes connected by the edge are a target node and a trajectory node respectively, take the perspective to which the target node belongs as the reference perspective; for the reference perspective, determine the feature similarity of the two nodes in the reference perspective according to the single-perspective features of the two nodes; for each perspective other than the reference perspective, determine the feature similarity of the two nodes in each other perspective according to the cross-perspective features of the two nodes; and determine the feature weight of the edge according to the feature similarity of each perspective.

[0105] Optionally, the single-view feature extraction network is trained by the following steps: obtaining image frames of a video of the same view at two preceding and succeeding moments, recording the image frame at the preceding moment as the preceding sample frame, and recording the image frame at the succeeding moment as the succeeding sample frame, wherein the preceding sample frame carries a true target label; performing target detection processing on the preceding sample frame to obtain target information of the target corresponding to the true target label as the preceding sample target information; using the single-view feature extraction network to be trained, extracting preceding sample single-view features from the preceding sample target information; performing target detection processing on the succeeding sample frame to obtain target information of multiple candidate frames as the succeeding sample target information of each candidate frame; using the single-view feature extraction network to be trained, extracting succeeding sample single-view features from the succeeding sample target information of each candidate frame; performing cross-correlation calculation on the preceding sample single-view features and the succeeding sample single-view features of each candidate frame to obtain a temporal cue matrix; determining a loss value based on the temporal cue matrix and the true target label; adjusting the parameters of the single-view feature extraction network to be trained based on the loss value to obtain a single-view feature extraction network.

[0106] Regarding the apparatus in the above embodiment, the specific manner in which each unit performs operations has been described in detail in the embodiment of the method, and will not be elaborated on here.

[0107] Figure 6 FIG. 6 shows a structural block diagram of an electronic device 600 according to an exemplary embodiment of the present disclosure.

[0108] Reference Figure 6 The electronic device 600 includes: at least one memory 601 and at least one processor 602, wherein the at least one memory 601 stores computer executable instructions. When the computer executable instructions are executed by the at least one processor 602, the at least one processor is prompted to execute the cross-view multi-target tracking method as described in the above exemplary embodiment.

[0109] As an example, electronic device 600 may be a PC, tablet device, personal digital assistant, smartphone, or other device capable of executing the aforementioned set of instructions. Here, electronic device 600 is not necessarily a single electronic device 600, but may also be any collection of devices or circuits capable of executing the aforementioned instructions (or instruction set) individually or in combination. Electronic device 600 may also be part of an integrated control system or system manager, or may be configured as a portable electronic device 600 that is interconnected locally or remotely (e.g., via wireless transmission) via an interface.

[0110] In electronic device 600, processor 602 may include a central processing unit (CPU), a graphics processing unit (GPU), a programmable logic device, a dedicated processor system, a microcontroller, or a microprocessor. By way of example and not limitation, processor 602 may also include an analog processor, a digital processor, a microprocessor, a multi-core processor, a processor array, a network processor, etc.

[0111] The processor 602 can execute instructions or codes stored in the memory 601, wherein the memory 601 can also store data. Instructions and data can also be sent and received over the network via the network interface device, wherein the network interface device can use any known transmission protocol.

[0112] The memory 601 may be integrated with the processor 602, for example, by placing RAM or flash memory within an integrated circuit microprocessor or the like. Furthermore, the memory 601 may comprise a separate device, such as an external disk drive, a storage array, or any other storage device usable by a database system. The memory 601 and the processor 602 may be operatively coupled or may communicate with each other, for example, via an I / O port, a network connection, or the like, such that the processor 602 can access files stored in the memory.

[0113] In addition, the electronic device 600 may further include a video display (such as a liquid crystal display) and a user interaction interface (such as a keyboard, a mouse, a touch input device, etc.) All components of the electronic device 600 may be connected to each other via a bus and / or a network.

[0114] According to an exemplary embodiment of the present disclosure, a computer-readable storage medium storing instructions may also be provided, wherein the instructions, when executed by at least one processor, prompt the at least one processor to perform the cross-view multi-target tracking method as described in the above exemplary embodiment. Examples of computer-readable storage media include: read-only memory (ROM), random access programmable read-only memory (PROM), electrically erasable programmable read-only memory (EEPROM), random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), flash memory, non-volatile memory, CD-ROM, CD-R, CD+R, CD-RW, CD+RW, DVD-ROM, DVD-R, DVD+R, DVD-RW, DVD+RW, DVD-RAM, BD-ROM, BD-R, BD-R LTH, BD-RE, Blu-ray or optical disk storage, hard disk drive (HDD), solid state drive (SSD), card storage (such as a multimedia card, secure digital (SD) card or extreme digital (XD) card), magnetic tape, floppy disk, magneto-optical data storage device, optical data storage device, hard disk, solid state disk and any other device configured to store a computer program and any associated data, data files and data structures in a non-transitory manner and provide the computer program and any associated data, data files and data structures to a processor or computer so that the processor or computer can execute the computer program. The computer program in the above-mentioned computer-readable storage medium can be executed in an environment deployed in a computer device such as a client, a host, an agent device, a server, etc. In addition, in one example, the computer program and any associated data, data files and data structures are distributed on a networked computer system so that the computer program and any associated data, data files and data structures are stored, accessed and executed in a distributed manner by one or more processors or computers.

[0115] According to an exemplary embodiment of the present disclosure, a computer program product may further be provided, including computer instructions, which, when executed by at least one processor, execute the cross-view multi-target tracking method as described in the above exemplary embodiment.

[0116] Other embodiments of the present disclosure will readily occur to those skilled in the art after considering the specification and practicing the invention disclosed herein. This disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not disclosed herein. The description and examples are to be considered as exemplary only, with the true scope and spirit of the present disclosure being indicated by the appended claims.

[0117] It should be understood that the present disclosure is not limited to the exact structures that have been described above and shown in the drawings, and that various modifications and changes can be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A cross-view multi-target tracking method, characterized in that: include: At each point in the tracking process, the following steps are performed: Perform target detection processing on the current frames of videos from different perspectives to obtain target information of each target in each perspective; Using a single-view feature extraction network, extracting single-view features from the target information; Extracting cross-view features from the target information using a cross-view feature extraction network, wherein the cross-view feature extraction network is used to extract similar features for the same target at different viewpoints; Constructing an undirected graph, wherein the nodes in the undirected graph include a target node for each target detected at each perspective at the current moment, and a trajectory node for each target trajectory determined at the previous moment, wherein the target trajectory is a motion path of the same target in a time series; Splitting nodes corresponding to the same target in the undirected graph into the same subgraph according to the attributes of each node, wherein the attributes include cross-view features and single-view features of each view involved in the corresponding node; Determine, based on the attributes of each node in each subgraph, a target trajectory corresponding to the subgraph and the attributes of the target trajectory; The undirected graph further includes an edge for connecting two nodes, and the attribute further includes location information. The nodes corresponding to the same target in the undirected graph are divided into the same subgraph according to the attribute of each node, including: For each edge in the undirected graph, determine the position distance between the two nodes connected by the edge based on the position information of the two nodes, as the position weight of the edge; determine the feature similarity between the two nodes based on at least one of the single-view feature and the cross-view feature of the two nodes connected by the edge, as the feature weight of the edge; determine the edge weight of the edge based on the position weight and the feature weight of the edge, wherein the edge weight is used to represent the possibility that the corresponding two nodes correspond to the same object; Based on the edge weight of each edge in the undirected graph, the undirected graph is divided into a plurality of disjoint subgraphs by solving a minimum multicut problem of the undirected graph.

2. The cross-view multi-target tracking method according to claim 1, wherein: Determining the edge weight of the edge according to the position weight and the feature weight of the edge includes: When the position distance between the two nodes is less than a distance threshold, determining a weighted sum of the position weight and the feature weight of the edge as the edge weight of the edge; When the position distance between the two nodes is greater than or equal to a distance threshold, a difference between the weighted sum value and a preset penalty value is determined as the edge weight of the edge.

3. The cross-view multi-target tracking method according to claim 1, wherein: The determining, based on at least one of the single-view feature and the cross-view feature of the two nodes connected by the edge, a feature similarity degree of the two nodes as the feature weight of the edge includes: When the two nodes connected by the edge are target nodes from different perspectives, determining the feature similarity between the two nodes according to the cross-perspective features of the two nodes as the feature weight of the edge; When the two nodes connected by the edge are respectively a target node and a trajectory node, the perspective to which the target node belongs is taken as a reference perspective; for the reference perspective, the feature similarity of the two nodes in the reference perspective is determined based on the single-perspective features of the two nodes; for each perspective other than the reference perspective, the feature similarity of the two nodes in each other perspective is determined based on the cross-perspective features of the two nodes; and the feature weight of the edge is determined based on the feature similarity of each perspective.

4. The cross-view multi-target tracking method according to any one of claims 1 to 3, wherein: The single-view feature extraction network is trained by the following steps: Obtain image frames of the video at the same perspective at two moments, record the image frame at the previous moment as the previous sample frame, and record the image frame at the next moment as the next sample frame, wherein the previous sample frame carries the real target label; Performing target detection processing on the prior sample frame to obtain target information of the target corresponding to the true target label as the prior sample target information; Using a single-view feature extraction network to be trained, extracting single-view features of the prior sample from the prior sample target information; Performing target detection processing on the subsequent sample frame to obtain target information of multiple candidate frames as subsequent sample target information of each candidate frame; Using the single-view feature extraction network to be trained, extracting subsequent sample single-view features from subsequent sample target information of each candidate box; Performing cross-correlation calculation on the single-view features of the previous sample and the single-view features of the subsequent sample of each candidate frame to obtain a temporal cue matrix; Determining a loss value according to the temporal cue matrix and the true target label; According to the loss value, the parameters of the single-view feature extraction network to be trained are adjusted to obtain the single-view feature extraction network.

5. A cross-view multi-target tracking device, characterized in that: include: At each moment during the trace, the following units are called: a detection unit configured to perform target detection processing on current frames of videos from different perspectives, respectively, to obtain target information of each target at each perspective; A first extraction unit is configured to extract a single-view feature from the target information using a single-view feature extraction network; a second extraction unit configured to extract cross-view features from the target information using a cross-view feature extraction network, wherein the cross-view feature extraction network is used to extract similar features for the same target at different views; a construction unit configured to construct an undirected graph, wherein the nodes in the undirected graph include a target node for each target detected at each perspective at the current moment, and a trajectory node for each target trajectory determined at the previous moment, wherein the target trajectory is a motion path of the same target in a time series; a segmentation unit configured to segment nodes corresponding to the same target in the undirected graph into the same subgraph according to attributes of each node, wherein the attributes include cross-view features and single-view features of each view involved in the corresponding node; a determining unit configured to determine a target trajectory corresponding to each subgraph and the attributes of the target trajectory based on the attributes of each node in each subgraph; The undirected graph further includes an edge for connecting two nodes, the attribute further includes location information, and the segmentation unit is further configured to: For each edge in the undirected graph, determine the position distance between the two nodes connected by the edge based on the position information of the two nodes, as the position weight of the edge; determine the feature similarity between the two nodes based on at least one of the single-view feature and the cross-view feature of the two nodes connected by the edge, as the feature weight of the edge; determine the edge weight of the edge based on the position weight and the feature weight of the edge, wherein the edge weight is used to represent the possibility that the corresponding two nodes correspond to the same object; Based on the edge weight of each edge in the undirected graph, the undirected graph is divided into a plurality of disjoint subgraphs by solving a minimum multicut problem of the undirected graph.

6. An electronic device, characterized in that: include: at least one processor; at least one memory storing computer-executable instructions, When the computer executable instructions are executed by the at least one processor, the at least one processor is prompted to execute the cross-view multi-target tracking method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that When the instructions in the computer-readable storage medium are executed by at least one processor, the instructions cause the at least one processor to perform the cross-view multi-target tracking method according to any one of claims 1 to 4.

8. A computer program product comprising computer instructions, characterized in that When the computer instructions are executed by at least one processor, the computer instructions cause the at least one processor to perform the cross-view multi-target tracking method according to any one of claims 1 to 4.

Images