A training method for a multi-target tracking model and a multi-target tracking method

Abstract

The embodiment of the present invention discloses a training method for a multi-target tracking model and a multi-target tracking method. The multi-target tracking method includes: constructing a target graph based on the target to be tracked in the current video frame; the vertices of the target graph correspond to the target to be tracked, and the edge features of the edges between two vertices include the attribute relationship between the two vertices; performing graph matching on the target graph and the existing trajectory graph to calculate the matching score between the target to be tracked and the tracked track in the trajectory graph; the vertices of the trajectory graph correspond to the existing tracked track, and the edge features of the edges between two vertices include the attribute relationship between the two vertices; determining the matching track of the target to be tracked according to the matching score. When matching the graph, the edge features in the target graph and the trajectory graph are fully referenced to prevent the problem of tracking failure when a target is blocked, avoid the limitations of video multi-target tracking, and improve the robustness of video multi-target tracking.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of video data processing technology, and in particular to a training method for a multi-target tracking model and a multi-target tracking method. Background Art

[0002] With the rapid development of computer vision and deep learning, video multi-target tracking algorithms have been widely used in fields such as intelligent surveillance, robotic vision systems, and virtual reality (VR) to focus on the motion trajectories of various target objects in the video over time.

[0003] At present, the existing multi-target tracking algorithms are mainly divided into two stages: target detection and target tracking. In the target detection stage, each target in each video frame will be detected frame by frame. Then, in the target tracking stage, the features of each target in each video frame will be extracted through a neural network model, and a bipartite graph matching algorithm will be used to associate the same target in consecutive video frames according to the feature similarity between the target in the previous video frame and the target in the next video frame in each adjacent video frame, thereby obtaining the motion trajectory of each target in the video.

[0004] However, this method cannot accurately match the trajectory of the target object if one target is occluded or there are too many targets, which makes video multi-target tracking have certain limitations and leads to low robustness of the video multi-target tracking algorithm. Therefore, a more robust multi-target tracking method is needed. Summary of the Invention

[0005] In view of this, an embodiment of the present invention provides a training method for a multi-target tracking model and a multi-target tracking method to achieve accurate tracking of multiple targets in a video, avoid the limitations of video multi-target tracking, and improve the robustness of video multi-target tracking.

[0006] In the first aspect, an embodiment of the present invention provides a training method for a multi-target tracking model, the method comprising: constructing an initial multi-target tracking model, the multi-target tracking model comprising a target graph extraction network, a trajectory graph extraction network and a graph matching network; the target graph extraction network is used to construct a target graph based on the target to be tracked in the current video frame, the trajectory graph extraction network is used to construct a trajectory graph based on the tracked trajectory in the historical video frame, and the graph matching network is used to perform graph matching on the target graph and the trajectory graph to predict the matching trajectory of the target to be tracked; wherein, both the target graph and the trajectory graph include vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked respectively, the vertices of the trajectory graph correspond to the tracked trajectory, and the edge features of the edges between two vertices in the target graph and the trajectory graph include the attribute relationship between the two vertices; using the preset loss function of the multi-target tracking model, the various networks in the multi-target tracking model are jointly trained according to the actual matching trajectory of the target to be tracked in the training sample and the predicted matching trajectory to obtain a trained multi-target tracking model.

[0007] In a second aspect, an embodiment of the present invention provides a multi-target tracking method, the method comprising: constructing a target graph based on the target to be tracked in the current video frame; wherein the target graph includes vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked, and the edge features of the edges between two vertices include the attribute relationship between the two vertices; performing graph matching on the target graph and an existing trajectory graph to calculate the matching score between the target to be tracked and the tracked track in the trajectory graph; wherein the trajectory graph includes vertices and edges connecting the vertices, the vertices of the trajectory graph correspond to the existing tracked track, and the edge features of the edges between two vertices include the attribute relationship between the two vertices; determining the matching trajectory of the target to be tracked according to the matching score.

[0008] In a third aspect, an embodiment of the present invention provides a training device for a multi-target tracking model, which includes: a model construction module for constructing an initial multi-target tracking model, the multi-target tracking model including a target graph extraction network, a trajectory graph extraction network and a graph matching network; the target graph extraction network is used to construct a target graph based on the target to be tracked in the current video frame, the trajectory graph extraction network is used to construct a trajectory graph based on the tracked trajectory in the historical video frame, and the graph matching network is used to perform graph matching on the target graph and the trajectory graph to predict the matching trajectory of the target to be tracked; wherein, both the target graph and the trajectory graph include vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked respectively, the vertices of the trajectory graph correspond to the tracked trajectory, and the edge features of the edges between two vertices in the target graph and the trajectory graph include the attribute relationship between the two vertices; a model training module is used to use the preset loss function of the multi-target tracking model to jointly train each network in the multi-target tracking model according to the actual matching trajectory of the target to be tracked in the training sample and the predicted matching trajectory to obtain a trained multi-target tracking model.

[0009] In a fourth aspect, an embodiment of the present invention provides a multi-target tracking device, which includes: a target graph construction module, used to construct a target graph based on the target to be tracked in the current video frame; wherein the target graph includes vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked, and the edge features of the edges between two vertices include the attribute relationship between the two vertices; a graph matching module, used to perform graph matching on the target graph and an existing trajectory graph to calculate the matching score between the target to be tracked and the tracked trajectory in the trajectory graph; wherein the trajectory graph includes vertices and edges connecting the vertices, the vertices of the trajectory graph correspond to the existing tracked trajectory, and the edge features of the edges between two vertices include the attribute relationship between the two vertices; a trajectory determination module, used to determine the matching trajectory of the target to be tracked according to the matching score.

[0010] In a fifth aspect, an embodiment of the present invention provides a computing device comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the training method of the multi-target tracking model in the first aspect above, or implement the multi-target tracking method in the second aspect above.

[0011] In a sixth aspect, an embodiment of the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the training method of the multi-target tracking model in the first aspect above, or implements the multi-target tracking method in the second aspect above.

[0012] An embodiment of the present invention provides a training method for a multi-target tracking model and a multi-target tracking method. When tracking the trajectories of each target to be tracked in a current video frame, each tracked trajectories has been determined for a historical video frame, and a corresponding trajectory graph has been constructed. Each vertex of the trajectory graph corresponds to each existing tracked trajectory, and the edge feature of the edge between two vertices includes the attribute relationship between the two vertices. At this time, a target graph is constructed according to each target to be tracked in the current video frame. Each vertex of the target graph corresponds to each target to be tracked, and the edge feature of the edge between two vertices includes the attribute relationship between the two vertices. Then, graph matching is performed on the target graph and the trajectory graph to calculate the matching score between each target to be tracked and each tracked trajectory in the trajectory graph, thereby determining the matching trajectory of each target to be tracked and realizing accurate tracking of multiple targets in the video.

[0013] During graph matching, the edge features in the target graph and trajectory graph are fully referenced to prevent tracking failure when a target is occluded, avoid the limitations of video multi-target tracking, and improve the robustness of video multi-target tracking. At the same time, if multi-target tracking is achieved through a multi-target tracking model, the multi-target tracking model needs to be trained first. At this time, the initially constructed multi-target tracking model includes a target graph extraction network, a trajectory graph extraction network, and a graph matching network, which are respectively used to execute the target graph construction, trajectory graph construction, and graph matching steps in the multi-target tracking process, and set a corresponding preset loss function for the multi-target tracking model. Then, using the preset loss function, according to the actual matching trajectory of each target to be tracked in the training sample and the matching trajectory predicted by the multi-target tracking model, the various networks in the multi-target tracking model are jointly trained to obtain the trained multi-target tracking model. There is no need to train each network in the multi-target tracking model separately, thereby improving the correlation between the networks in the multi-target tracking model and the accuracy of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1A A flowchart of a method for training a multi-target tracking model provided in Example 1 of the present invention;

[0015] Figure 1B A schematic diagram illustrating the principle of the training process of the multi-target tracking model provided in the first embodiment of the present invention;

[0016] Figure 2A A flowchart of a method for training a multi-target tracking model provided in the second embodiment of the present invention;

[0017] Figure 2B A schematic diagram illustrating the principle of a specific training process of a multi-target tracking model in the method provided in the second embodiment of the present invention;

[0018] Figure 3A flowchart of a multi-target tracking method provided in Example 3 of the present invention;

[0019] Figure 4A A flowchart of a multi-target tracking method provided in Example 4 of the present invention;

[0020] Figure 4B A schematic diagram illustrating the principle of the graph matching process in the method provided in the fourth embodiment of the present invention;

[0021] Figure 5 A flowchart of a multi-target tracking method provided in Example 5 of the present invention;

[0022] Figure 6 A schematic diagram of the structure of a multi-target tracking model training device provided in Example 6 of the present invention;

[0023] Figure 7 A schematic structural diagram of a multi-target tracking device provided in Embodiment 7 of the present invention;

[0024] Figure 8 A schematic diagram of the structure of a computing device provided in Example 8 of the present invention. DETAILED DESCRIPTION

[0025] The present invention will be further described in detail below with reference to the accompanying drawings and examples. It will be understood that the specific embodiments described herein are intended only to illustrate the present invention and are not intended to limit the present invention. It should also be noted that, for ease of description, the accompanying drawings only illustrate portions relevant to the present invention, not all of the structures. Furthermore, the embodiments of the present invention and the features of the embodiments may be combined with one another unless otherwise specified.

[0026] Example 1

[0027] Figure 1A This is a flowchart of a training method for a multi-target tracking model provided in Example 1 of the present invention. This embodiment can be applied to the situation where multi-target tracking is performed on any video. The training method for a multi-target tracking model provided in this embodiment can be performed by a training device for a multi-target tracking model provided in an embodiment of the present invention. The device can be implemented in software and / or hardware and integrated in a computing device that executes this method.

[0028] Specifically, refer to Figure 1A , the method may include the following steps:

[0029] S110, constructing an initial multi-target tracking model.

[0030] Specifically, in order to achieve accurate tracking of multiple targets in a video, a neural network model is usually trained to continuously perform corresponding multi-target tracking operations on each video frame, so that the trained neural network model can accurately output the matching trajectory of each target after tracking in any video frame; at this time, when performing multi-target tracking on any video frame in a certain video, it can be divided into the current video frame that needs to perform multi-target tracking and the historical video frame that has completed multi-target tracking, that is, there are multiple targets to be tracked in the current video frame, and by analyzing the position of each target in different historical video frames, the tracking of the target in the historical video frame can be achieved, thereby determining multiple tracked trajectories. At this time, performing multi-target tracking on the current video frame is to match each target to be tracked in the current video frame with each tracked trajectories under the historical video frame, so as to determine the matching trajectory of each target to be tracked from each tracked trajectories.

[0031] In this embodiment, for the target to be tracked in the current video frame, each target to be tracked will be taken as a corresponding vertex, and then the vertices will be connected to construct a corresponding target graph. At this time, the target graph includes vertices and edges connecting the vertices. Each vertex in the target graph corresponds to each target to be tracked in the current video frame, and the vertex feature of each vertex is the attribute feature of the target to be tracked corresponding to the vertex. The attribute feature may include various features that can describe the unique characteristics of the target to be tracked in the current video frame, such as the appearance feature, geometric shape feature, and trajectory speed of the target to be tracked. At the same time, the edge feature of the edge connecting two vertices in the target graph may include the attribute relationship between the two vertices, that is, the relationship between the attribute features of the two vertices.

[0032] At the same time, for the tracked trajectories in the historical video frames, each tracked trajectory will be taken as a corresponding vertex, and then the vertices will be connected to construct a corresponding trajectory graph. At this time, the trajectory graph also includes vertices and edges connecting the vertices. Each vertex in the trajectory graph corresponds to each tracked trajectory that has been tracked in the historical video frame. The tracked trajectory is connected by the positions of the same target in different historical video frames. The vertex feature of each vertex in the trajectory graph is the attribute feature of the tracked trajectory corresponding to the vertex, and the attribute feature of the tracked trajectory can be obtained by fusing the attribute features of the targets corresponding to the tracked trajectory in different historical video frames. For example, the attribute features of the tracked trajectory can be obtained by averaging the attribute features of the targets corresponding to each tracked trajectory in different historical video frames. In addition, the edge feature of the edge connecting two vertices in the trajectory graph can also include the attribute relationship between the two vertices, that is, the relationship between the attribute features of the two vertices.

[0033] At this time, the vertex features and edge features in the target graph can fully describe each target to be tracked in the current video frame, and the vertex features and edge features in the trajectory graph can also fully describe each tracked trajectory in the historical video frame. Therefore, by performing graph matching on the target graph and the trajectory graph, the matching trajectory of each target to be tracked can be determined from each tracked trajectory.

[0034] In other words, the execution steps for video multi-target tracking in this embodiment can be mainly divided into three steps: constructing a target map of the target to be tracked in the current video frame, constructing a trajectory map of the tracked trajectories in the historical video frame, and performing graph matching on the target map and the trajectory map. Therefore, in order to ensure the execution accuracy of each step in video multi-target tracking, when constructing the initial multi-target tracking model, this embodiment can set up corresponding target map extraction network, trajectory map extraction network and graph matching network in the multi-target tracking model. Among them, the target map extraction network is used to construct a target map based on the target to be tracked in the current video frame, the trajectory map extraction network is used to construct a trajectory map based on the tracked trajectories in the historical video frame, and the graph matching network is used to perform graph matching on the constructed target map and the trajectory map, thereby predicting the matching trajectory of each target to be tracked from each tracked trajectory, so that each network in the multi-target tracking model can specifically execute a specific step corresponding to the video multi-target tracking process, thereby ensuring the accuracy of video multi-target tracking.

[0035] It should be noted that the target to be tracked in this embodiment can be a target object whose position changes continuously in different video frames of a certain video, such as pedestrians, vehicles, animals, etc., which need to be tracked. This is not limited to this. Each target to be tracked can be identified from the current video frame through a specific target detection algorithm. At the same time, in order to ensure the efficient and convenient use of vertex features and edge features in the target graph and trajectory graph in the subsequent graph matching process, in this embodiment, the attribute features of the vertices of the target graph and trajectory graph can be represented as vector features, and the edge features of the edge between two vertices can also be represented as a spliced vector or weighted vector of the vector features corresponding to the two vertices.

[0036] In addition, to ensure the comprehensiveness of the reference features used when matching the target graph with the trajectory graph, in this embodiment, both the target graph and the trajectory graph can be set as complete graphs, that is, an edge is connected between every two vertices of the target graph and the trajectory graph to associate the target to be tracked corresponding to every two vertices in the target graph or the tracked trajectory corresponding to every two vertices in the trajectory graph, so as to facilitate the subsequent accurate graph matching of the target graph and the trajectory graph. Of course, those skilled in the art can also set the target graph and the trajectory graph as incomplete graphs as needed, that is, allow some vertices to have no edge connection.

[0037] S120, using the preset loss function of the multi-target tracking model, jointly train each network in the multi-target tracking model according to the actual matching trajectory and the predicted matching trajectory of the target to be tracked in the training sample to obtain a trained multi-target tracking model.

[0038] Optionally, after constructing the initial multi-target tracking model, it is necessary to perform multi-target tracking training on the multi-target tracking model so that the multi-target tracking model can accurately track the trajectory of the target to be tracked in any video. At this time, when training the constructed multi-target tracking model, a corresponding preset loss function will first be set for the multi-target tracking model. The preset loss function is used to determine the difference between the predicted results and the actual results after multi-target tracking in the current training process, so as to continuously adjust various model parameters in the multi-target tracking model.

[0039] Specifically, a large number of historical videos containing multi-target tracking content are first obtained as training samples for the multi-target tracking model. At this time, for each training sample, the true matching trajectory of each target to be tracked in each video frame in the training sample will be marked. After continuously predicting the matching trajectory of the target to be tracked in each video frame in the training sample during the model training process, the difference between the true matching trajectory of each target to be tracked and the predicted matching trajectory is compared through a preset loss function to adjust the multi-target tracking model. In other words, the historical video has the tracked trajectory of each target in the video frame. Each frame in the historical video can be used as the current frame, and each current frame is marked with the matching trajectory of the target and the tracked trajectory of the historical video.

[0040] At this time, the current video frame in the training sample is input into the target graph extraction network in the constructed multi-target tracking model. The target graph extraction network identifies each target to be tracked in the current video frame and detects the attribute features of each target to be tracked. Then, each target to be tracked is used as a vertex and each vertex is connected to construct a corresponding target graph. At the same time, the vertex features of each vertex in the target graph are determined, and the edge features of the edge connecting the two vertices are determined according to the attribute relationship between the two vertices for use in the subsequent graph matching process.

[0041] At the same time, the historical video frames in the training sample are input into the trajectory graph extraction network within the constructed multi-target tracking model. At this point, each historical video frame has completed multi-target tracking, that is, the trajectory graph extraction network can identify the various tracked trajectories formed after tracking multiple identical targets in different historical video frames, and by performing feature fusion (such as feature mean processing, etc.) on the attribute features of the same target in each historical video frame, the attribute features of the tracked trajectory corresponding to the target are obtained. Then, each tracked trajectory is used as a vertex, and each vertex is connected to construct the corresponding trajectory graph. At the same time, the vertex features of each vertex in the trajectory graph are determined, and the edge features of the edge connecting the two vertices are determined according to the attribute relationship between the two vertices for use in the subsequent graph matching process.

[0042] The target graph and trajectory graph are then input into the graph matching network within the constructed multi-target tracking model. The graph matching network calculates the vertex similarity and edge similarity between the target graph and the trajectory graph based on the vertex features of each vertex and the edge features of each edge in the target and trajectory graphs. Since the vertices in the target graph correspond to the targets to be tracked, and the vertices in the trajectory graph correspond to the tracked trajectories, the matching trajectory of each target to be tracked is determined from the tracked trajectories, that is, each vertex in the target graph and the trajectory graph is matched. Therefore, for each vertex in the target graph, the graph matching network can comprehensively analyze the overall matching degree between the vertex and each vertex in the trajectory graph based on the vertex features of the vertex in the target graph and the edge features of each edge associated with the vertex in the trajectory graph, as well as the vertex features of each vertex in the trajectory graph and the edge features of each edge associated with each vertex in the trajectory graph. Even if a vertex in the target graph is partially occluded, the matching degree between the vertex and each vertex in the trajectory graph can be analyzed using the associated edge features, thus preventing tracking failure when a target is occluded and overcoming the limitations of video multi-target tracking.

[0043] That is, the graph matching network can predict the matching trajectory of each target to be tracked in the current video frame of the training sample from each tracked trajectory in the historical video frame in the training sample; then, the actual matching trajectory of each target to be tracked in the training sample and the predicted matching trajectory of the target to be tracked are substituted into the preset loss function of the multi-target tracking model, and the difference between the matching trajectory predicted and the actual matching trajectory in the current training process is analyzed by the preset loss function, and then according to the reverse process of the multi-target tracking process, the difference is back-propagated in the graph matching network, trajectory graph extraction network and target graph extraction network in the multi-target tracking model to synchronously correct various network parameters in the graph matching network, trajectory graph extraction network and target graph extraction network, so that the difference between the matching trajectory predicted and the actual matching trajectory in the current training process continues to converge, thereby jointly training the various networks in the multi-target tracking model, and finally obtaining the trained multi-target tracking model, thereby improving the correlation between the networks in the multi-target tracking model and the accuracy of multi-target tracking.

[0044] The technical solution provided by this embodiment is that if multi-target tracking is achieved through a multi-target tracking model, the multi-target tracking model needs to be trained first. At this time, the initially constructed multi-target tracking model includes a target graph extraction network, a trajectory graph extraction network and a graph matching network, which are respectively used to perform the target graph construction, trajectory graph construction and graph matching steps in the multi-target tracking process, and set a corresponding preset loss function for the multi-target tracking model. Then, using the preset loss function, according to the actual matching trajectory of each target to be tracked in the training sample and the matching trajectory predicted by the multi-target tracking model, the various networks in the multi-target tracking model are jointly trained to obtain the trained multi-target tracking model. There is no need to specifically train each network in the multi-target tracking model separately, thereby improving the correlation between the networks in the multi-target tracking model and the accuracy of multi-target tracking.

[0045] Example 2

[0046] Figure 2A This is a flowchart of a method for training a multi-target tracking model provided in Example 2 of the present invention. Figure 2B This is a schematic diagram of the principle of the specific training process of the multi-target tracking model in the method provided in the second embodiment of the present invention. This embodiment is an optimization based on the above embodiment. Specifically, Figure 2BAs shown, on the basis of the above technical solution, in order to improve the accuracy of the graph matching network in matching the target graph and the trajectory graph according to the vertex similarity and edge similarity between the target graph and the trajectory graph, this embodiment will also set up an additional feature enhancement network in the multi-target tracking model when constructing the initial multi-target tracking model. The feature enhancement network is used to enhance the vertex features and edge features in the constructed target graph and trajectory graph after constructing the target graph of the target to be tracked in the current video frame through the target graph extraction network and constructing the trajectory graph of the tracked trajectory in the historical video frame through the trajectory graph extraction network, so that the graph matching network can perform graph matching on the target graph and the trajectory graph according to the enhanced vertex features and edge features in the target graph and the trajectory graph.

[0047] Specifically, such as Figure 2B As shown, this embodiment may include the following steps:

[0048] S210, constructing an initial multi-target tracking model.

[0049] S220, using the target map extraction network in the multi-target tracking model to construct a target map of the target to be tracked in the current video frame in the training sample.

[0050] Optionally, the current video frame in the training sample is input into the target graph extraction network within the constructed multi-target tracking model, and the target graph extraction network is used to identify each target to be tracked in the current video frame, and detect the attribute characteristics of each target to be tracked. Then, each target to be tracked is used as a vertex, and each vertex is connected to construct a corresponding target graph. At the same time, the vertex characteristics of each vertex in the target graph are determined, and the edge characteristics of the edge connecting the two vertices are determined according to the attribute relationship between the two vertices.

[0051] S230, using a trajectory graph extraction network in the multi-target tracking model to construct a trajectory graph of tracked trajectories in historical video frames in the training sample.

[0052] Optionally, the historical video frames in the training sample are input into the trajectory graph extraction network in the constructed multi-target tracking model to obtain the attribute features of the tracked trajectory corresponding to the target, and then each tracked trajectory is used as a vertex and each vertex is connected to construct the corresponding trajectory graph. At the same time, the vertex features of each vertex in the trajectory graph are determined, and the edge features of the edge connecting the two vertices are determined according to the attribute relationship between the two vertices.

[0053] S240, using the feature enhancement network in the multi-target tracking model to enhance the vertex features and edge features in the target graph and the trajectory graph, and inputting the enhanced vertex features and edge features in the target graph and the trajectory graph into the graph matching network in the multi-target tracking model to predict the matching trajectory of the target to be tracked.

[0054] In this embodiment, to further improve the matching accuracy between vertices in the target graph and the trajectory graph, the vertex features of each vertex in the target graph and the trajectory graph, as well as the edge features of the edges connecting the two vertices, are first input into the feature enhancement network within the multi-target tracking model. This feature enhancement network then analyzes the feature similarity between each vertex in the target graph and each vertex in the trajectory graph. The feature similarity is then used as the weight between the vertex and each vertex in the trajectory graph. Using the weights between the vertex and each vertex in the trajectory graph, the vertex features of each vertex in the trajectory graph are weightedly fused into the vertex features of the vertex, thereby obtaining the vertex features of the vertex after feature enhancement.

[0055] For example, if the vertex feature of a vertex in the target graph is h i , the vertex feature of each vertex in the trajectory graph is h j,1 、h j,2 ,…,h j,n , where n is the number of vertices in the trajectory graph. At this time, the feature similarity w between the vertex feature of each vertex in the trajectory graph and the vertex feature of the vertex in the target graph can be calculated i,j1 、w i,j2 、…、w i,jn At this time, the vertex feature of the vertex in the target graph after feature enhancement can be: h i '=h i +w i,j1 ·h j,1 +w i,j2 ·h j,2 +L+w i,jn ·h j,n , so that the vertex features of a certain vertex after feature enhancement are more similar to the vertex features of similar vertices in the trajectory graph, and are easier to distinguish from the vertex features of dissimilar vertices in the trajectory graph. According to the same steps as above, the feature similarity between each vertex in the target graph and each vertex in the trajectory graph can be calculated to enhance the vertex features of each vertex in the target graph using the feature weighting results of each vertex in the trajectory graph. At the same time, for the vertex features of each vertex in the trajectory graph, the above-mentioned vertex feature enhancement method in the target graph can be adopted. By analyzing the feature similarity between the vertex and each vertex in the target graph as the corresponding weight, the vertex features of each vertex in the target graph are weighted and fused into the vertex features of the vertex using the weights between the vertex and each vertex in the target graph, thereby obtaining the vertex features of the vertex in the trajectory graph after feature enhancement.

[0056] In one implementation, the enhanced vertex features may be directly concatenated or weighted to serve as enhanced edge features.

[0057] In another implementation, for the edge features of each edge in the target graph and the trajectory graph, the same method as the vertex feature enhancement provided above can be used to calculate the feature similarity between the edge features of each edge in the target graph and the edge features of each edge in the trajectory graph, respectively, as the weight between each edge in the target graph and each edge in the trajectory graph. Then, using this weight, the edge features of each edge in the trajectory graph are weighted and fused into the edge features of the edge of the target graph, or the edge features of each edge in the target graph are weighted and fused into the edge features of the edge of the trajectory graph, thereby enhancing the edge features of each edge in the target graph and the trajectory graph, and obtaining the edge features of each edge in the target graph and the trajectory graph after feature enhancement. Subsequently, after calculating the vertex similarity and edge similarity between the target graph and the trajectory graph using the vertex features and edge features after feature enhancement in the target graph and the trajectory graph through the graph matching network, the matching trajectory of each target to be tracked can be more accurately predicted.

[0058] As an optional method for weight calculation in the above scheme, this embodiment can use the following formula to calculate the weight between each vertex in the target graph and each vertex in the trajectory graph, or to calculate the weight between each edge in the target graph and each edge in the trajectory graph: i.j =cos(h i ,h j )+IoU(g i ,g j ). If we calculate the weights between the vertices in the target graph and the trajectory graph, then h i is the vertex feature of a vertex in the target graph, h j is the vertex feature of a vertex in the trajectory graph, g i is the size of the detection box of the target to be tracked corresponding to the vertex in the target graph in the current video frame, g j is the detection box size of the target in the historical video frame corresponding to the tracked trajectory of the vertex in the trajectory graph, cos(h i ,h j ) represents the feature similarity between the vertex in the target graph and the vertex in the trajectory graph, IoU(g i ,g j ) represents the overlap ratio of detection frames between vertices in the target graph and the trajectory graph, that is, the ratio of the intersection and union of detection frames. If the weights between the edges in the target graph and the trajectory graph are calculated, then h i is the edge feature of an edge in the target graph, h j is the edge feature of a certain edge in the trajectory graph, g i is the size of the detection box of the target to be tracked in the current video frame corresponding to the two vertices connected by the edge in the target graph, g jis the detection box size of the relevant target in the historical video frame corresponding to the two vertices connected by the edge in the trajectory graph, cos(h i ,h j ) represents the feature similarity between the edge in the target graph and the edge in the trajectory graph, IoU(g i ,g j ) represents the overlap ratio of detection frames between two vertices connected by edges in the target graph and the trajectory graph, that is, the ratio of the intersection and union of the two detection frames, so as to ensure the accuracy of the weights selected for feature enhancement, thereby improving the accuracy of feature enhancement of vertex features and edge features in the target graph and the trajectory graph.

[0059] It should be noted that the feature enhancement network in this embodiment can be a cross-graph graph convolutional network or an information transfer network or other graph neural network with feature aggregation and enhancement functions.

[0060] Furthermore, after the enhanced vertex features and edge features in the target graph and trajectory graph are input into the graph matching network within the multi-target tracking model, the graph matching network can calculate the vertex similarity and edge similarity between the target graph and the trajectory graph based on the enhanced vertex features and edge features in the target graph and the trajectory graph, calculate the matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph based on the vertex similarity and edge similarity, and predict the matching trajectory of the target to be tracked based on the matching score. Specifically, the graph matching network first uses the corresponding enhanced vertex features and edge features for each vertex and edge in the target graph, respectively, and the vertex features of each vertex and edge features in the trajectory graph to calculate the vertex similarity between each vertex in the target graph and each vertex in the trajectory graph, as well as the edge similarity between each edge in the target graph and each edge in the trajectory graph. Then, for each vertex corresponding to the target to be tracked in the target graph, the vertex similarity between that vertex and each vertex in the trajectory graph, as well as the edge similarity between each edge connecting that vertex in the target graph and each edge in the trajectory graph, is analyzed to determine the overall degree of match between each vertex in the target graph and each vertex in the trajectory graph. This results in a matching score between the target to be tracked in the target graph and the tracked tracks in the track graph. The matching track of each target to be tracked is then predicted from each tracked track by finding the highest matching score between each target to be tracked and each tracked track.

[0061] On the basis of the above technical solution, Figure 2BAs shown, the graph matching network in this embodiment can be composed of a vertex similarity conversion layer, an edge similarity conversion layer and a quadratic programming layer. The vertex similarity conversion layer is used to calculate the vertex similarity between the target graph and the trajectory graph and perform dimensionality conversion. The edge similarity conversion layer is used to calculate the edge similarity between the target graph and the trajectory graph and perform dimensionality conversion. The quadratic programming layer is used to calculate the matching score based on the quadratic programming problem using the vertex similarity and edge similarity after dimensionality conversion.

[0062] Specifically, after the vertex features of each vertex and edge in the target graph, as well as the vertex features of each vertex and edge in the trajectory graph, are input into the graph matching network, the vertex similarity conversion layer in the graph matching network analyzes the feature similarity between each vertex in the target graph and each vertex in the trajectory graph, thereby calculating the vertex similarity between the target graph and the trajectory graph. This vertex similarity is an m1×n1 matrix, where m1 is the number of vertices in the target graph and n1 is the number of vertices in the trajectory graph. The edge similarity conversion layer in the graph matching network analyzes the feature similarity between each edge in the target graph and each edge in the trajectory graph, thereby calculating the edge similarity between the target graph and the trajectory graph. This edge similarity is an m2×n2 matrix, where m2 is the number of edges in the target graph and n2 is the number of edges in the trajectory graph. Because the vertex similarity and edge similarity between the target graph and the trajectory graph have different dimensions, the vertex similarity and edge similarity cannot be quickly integrated when calculating the matching score between the target graph and the trajectory graph based on these two similarities. Therefore, in this embodiment, the vertex similarity conversion layer performs dimension conversion on the vertex similarity between the target graph and the trajectory graph, and the edge similarity conversion layer performs dimension conversion on the edge similarity between the target graph and the trajectory graph, so that the converted vertex similarity and edge similarity can be accurately and efficiently integrated. Furthermore, since the number of edges in the target graph and the trajectory graph is specifically determined by the number of vertices in the target graph and the trajectory graph, the vertex similarity conversion layer and the edge similarity conversion layer respectively set the corresponding conversion dimension according to the number of vertices in the target graph and the trajectory graph.

[0063] Furthermore, the quadratic programming layer will preliminarily relax the discrete allocation objective in the quadratic assignment problem used in the original graph matching process into a corresponding continuous programming objective, and then perform a corresponding transformation on the adjusted quadratic assignment problem to obtain the corresponding quadratic programming problem, so that the quadratic programming layer has differentiable and learnable functions for subsequent neural network training. For example, the quadratic programming problem in the quadratic programming layer can be: Among them, D' is the continuous planning target after relaxation adjustment, M is the edge similarity after dimension conversion, b is the vertex similarity after dimension conversion, X *is the matching score between the target to be tracked in the target graph and the tracked trajectories in the trajectory graph. The quadratic programming layer then substitutes the dimensionally converted vertex similarity and edge similarity into the quadratic programming problem to calculate the matching score between each target to be tracked in the target graph and each tracked trajectories in the trajectory graph. Based on this matching score, the matching trajectory of each target to be tracked is predicted from each tracked trajectories.

[0064] S250, using the preset loss function of the multi-target tracking network, based on the actual matching trajectory and predicted matching trajectory of each target to be tracked in the current video frame in the training sample in the historical video frame, the gradient backpropagation of the graph matching network, the feature enhancement network, the trajectory graph extraction network and the target graph extraction network is performed to jointly train the networks in the multi-target tracking model.

[0065] Optionally, after predicting the matching trajectory of each target to be tracked, the actual matching trajectory of each target to be tracked in the training sample is found, and then the actual matching trajectory of each target to be tracked and the predicted matching trajectory are substituted into the preset loss function of the multi-target tracking model to analyze the difference between the predicted matching trajectory and the actual matching trajectory under the current training. The preset loss function in this embodiment can be a weighted binary cross entropy loss function, and the weights in the preset loss function are set according to the number of vertices in the target graph and the trajectory graph, such as Among them, k is the weight in the preset loss function, y i,j is the true matching score of the target to be tracked, y' i,j The predicted matching score of the target to be tracked is calculated using the preset loss function to determine the difference between the predicted matching trajectory of the target to be tracked and the actual matching trajectory. This difference is then used to perform gradient backpropagation through the graph matching network, feature enhancement network, trajectory graph extraction network, and target graph extraction network within the multi-target tracking model, following the reverse tracking process of the multi-target tracking process. This allows for the simultaneous correction of various network parameters within these networks, allowing for joint training of the networks within the multi-target tracking model, enabling the model to accurately predict the matching trajectory of each target to be tracked.

[0066] The technical solution provided in this embodiment converts the quadratic assignment problem used in the original graph matching process into a quadratic programming problem, so that the graph matching network has differentiable and learnable functions, thereby jointly training the various networks in the multi-target tracking model, improving the correlation between the networks in the multi-target tracking model and the accuracy of multi-target tracking.

[0067] Example 3

[0068] Figure 3This is a flowchart of a multi-target tracking method provided in Example 3 of the present invention. This embodiment is applicable to performing multi-target tracking on any video. The multi-target tracking method provided in this embodiment can be performed by a multi-target tracking device provided in an embodiment of the present invention. This device can be implemented in software and / or hardware and integrated into the computing device performing this method.

[0069] Specifically, refer to Figure 3 , the method may include the following steps:

[0070] S310: Construct a target map according to the target to be tracked in the current video frame.

[0071] Specifically, in the video multi-target tracking scenario, the trajectory of each target in each video frame in the video will be tracked in turn. At this time, the video can be divided into the current video frame in which multi-target tracking is being performed and the historical video frame in which multi-target tracking has been completed. By analyzing the multi-target tracking completed in the historical video frame, multiple tracked trajectories can be determined according to the tracking position of the same target in each historical video frame; at this time, multi-target tracking of the current video frame is to match each target to be tracked in the current video frame with each tracked trajectories in the historical video frame, so as to determine the matching trajectory of each target to be tracked from each tracked trajectories.

[0072] In this embodiment, a preset target detection algorithm can be used to identify each target to be tracked in the current video frame, and then each target to be tracked is used as a corresponding vertex, and each vertex is connected to construct a corresponding target graph. The target graph includes vertices and edges connecting the vertices. Each vertex in the target graph corresponds to each target to be tracked, so that the vertex feature of each vertex in the target graph is the attribute feature of the target to be tracked corresponding to the vertex. At the same time, the edge feature of the edge between two vertices in the target graph can include the attribute relationship between the two vertices, that is, the relationship between the attribute features of the two vertices, such as the distance, appearance difference, or relative trajectory speed of the targets to be tracked corresponding to the two vertices in the current video frame.

[0073] Exemplarily, in this embodiment, a target graph is constructed based on the target to be tracked in the current video frame. Specifically, the target graph is extracted from the attribute features of each target to be tracked in the current video frame; each target to be tracked is used as a vertex to construct a target graph, and vertex features of each vertex in the target graph are obtained. Specifically, the target graph is constructed by analyzing the appearance features, geometric shape features, trajectory speed, and other features that can describe the unique characteristics of the target to be tracked in the current video frame to obtain the attribute features of the target to be tracked. Then, each target to be tracked is used as a vertex, and the vertices are connected to construct a target graph. In this case, the vertex features of each vertex in the target graph are the attribute features of the target to be tracked corresponding to that vertex.

[0074] S320 , performing graph matching on the target graph and the existing trajectory graph to calculate a matching score between the target to be tracked and the tracked trajectory in the trajectory graph.

[0075] Since the historical video frames have already completed multi-target tracking when the current video frame is tracked, multiple tracked trajectories can be determined according to the tracking position of the same target in each historical video frame. Therefore, after completing the multi-target tracking of the historical video frame, each tracked trajectory under the historical video frame will also be used as a corresponding vertex, and each vertex will be connected to construct a corresponding trajectory graph. At this time, the trajectory graph also includes vertices and edges connecting vertices. Each vertex in the trajectory graph corresponds to each existing tracked trajectory. The tracked trajectory is formed by connecting the positions of the same target in different historical video frames. The vertex feature of each vertex in the trajectory graph is the attribute feature of the tracked trajectory corresponding to the vertex. The attribute feature is obtained by fusion of the attribute features of the target corresponding to the tracked trajectory in each historical video frame. At the same time, the edge feature of the edge between two vertices in the trajectory graph can also include the attribute relationship between the two vertices, that is, the relationship between the attribute features of the two vertices.

[0076] It should be noted that, in this embodiment, the target graph and the trajectory graph can be set as incomplete graphs or complete graphs. In the latter, an edge will be connected between every two vertices of the target graph and the trajectory graph to associate the targets to be tracked corresponding to every two vertices in the target graph or the tracked trajectories corresponding to every two vertices in the trajectory graph, so as to facilitate the subsequent accurate graph matching of the target graph and the trajectory graph.

[0077] Specifically, by performing graph matching on the target graph and the trajectory graph, the degree of matching between each vertex in the target graph and each vertex in the trajectory graph can be analyzed, thereby calculating the matching score between each target to be tracked in the target graph (that is, each vertex in the target graph) and each tracked trajectory in the trajectory graph (that is, each vertex in the trajectory graph), so that the matching trajectory of each target to be tracked can be predicted from each tracked trajectory according to the matching score.

[0078] S330: Determine a matching trajectory of the target to be tracked according to the matching score.

[0079] Optionally, after calculating the matching score between each target to be tracked in the target map and each tracked track in the track map, for each target to be tracked, the tracked track with the highest matching score with the target to be tracked can be found, and then the tracked track can be used as the matching track of the target to be tracked; in this way, the matching track of each target to be tracked can be determined.

[0080] Furthermore, to ensure the accuracy of subsequent multi-target tracking, after determining the matching trajectory of each target to be tracked in the current video frame according to the matching score, the trajectory map is updated according to the matching trajectory of the target to be tracked in the current video frame, and the vertex features of each vertex in the trajectory map are updated according to the attribute features of the matching trajectory of each target to be tracked in the current video frame. In other words, the corresponding trajectory map is reconstructed with each tracked trajectory re-determined after completing multi-target tracking in the current video frame as the vertex, and by analyzing the attribute features of each target to be tracked in the current video frame, the attribute features of each tracked trajectory determined by the matching trajectory of each target to be tracked in the current video frame are re-determined, thereby correspondingly updating the vertex features of each vertex in the trajectory map, ensuring the accuracy of subsequent video frames during multi-target tracking.

[0081] The technical solution provided by this embodiment, when tracking the trajectory of each target to be tracked in the current video frame, determines each tracked trajectory for historical video frames and constructs a corresponding trajectory graph. Each vertex in this trajectory graph corresponds to an existing tracked trajectory, and the edge features of the edge between two vertices include the attribute relationship between the two vertices. At this time, a target graph is constructed based on each target to be tracked in the current video frame. Then, a graph matching is performed between the target graph and the trajectory graph to calculate the matching score between each target to be tracked and each tracked trajectory in the trajectory graph. This determines the matching trajectory of each target to be tracked, achieving accurate tracking of multiple targets in the video.

[0082] Example 4

[0083] Figure 4A This is a flowchart of a multi-target tracking method provided by the fourth embodiment of the present invention. Figure 4B This is a schematic diagram of the principle of the image matching process in the method provided in the fourth embodiment of the present invention. This embodiment is an optimization based on the above embodiment. Specifically, Figure 4B As shown, this embodiment can use the multi-target tracking model trained in the multi-target tracking model training method provided in the above embodiment to execute the specific multi-target tracking process.

[0084] Optional, such as Figure 4AAs shown, this embodiment may include the following steps:

[0085] S410: Construct a trajectory map according to the tracked trajectories in the historical video frames.

[0086] Optionally, in a video multi-target tracking scenario, when performing multi-target tracking on the current video frame, it is first necessary to analyze the tracking position of the same target in each historical video frame according to the multi-target tracking situation in each historical video frame that has completed multi-target tracking, and then determine each tracked track in the historical video frame, and use each tracked track as the corresponding vertex, connect each vertex, and construct the corresponding track map. At this time, if Figure 4B As shown, the historical video frame can be input into a trajectory graph extraction network in a pre-trained multi-target tracking model, and the trajectory graph extraction network is used to construct a trajectory graph of the tracked trajectories in the historical video frame.

[0087] Exemplarily, in order to accurately obtain the vertex features of each vertex in the trajectory graph, when constructing the trajectory graph, this embodiment will determine the associated target of each tracked trajectory in each historical video frame; extract the attribute sub-features of the associated target of the tracked trajectory in each historical video frame, and calculate the corresponding attribute sub-feature mean as the attribute feature of the tracked trajectory; construct a trajectory graph with each tracked trajectory as a vertex, and obtain the vertex feature of each vertex in the trajectory graph.

[0088] In this case, the associated target of each tracked trajectory in each historical video frame is the same target, and the positions of the same target in each historical video frame can be connected to form the tracked trajectory. In this case, by using a preset feature extraction algorithm, the attribute sub-features of each associated target pointed to by the tracked trajectory in each historical video frame can be accurately extracted. Then, the attribute sub-features of each associated target pointed to by each tracked trajectory are averaged to obtain the attribute characteristics of the tracked trajectory. For example, in this embodiment, a sliding average method can be used to calculate the corresponding attribute sub-feature mean based on the attribute sub-features of the associated targets of each tracked trajectory in each historical video frame, and then the attribute sub-feature mean is used as the attribute characteristics of the tracked trajectory, ensuring the accuracy and comprehensiveness of the attribute characteristics of the tracked trajectory. Then, after obtaining the attribute characteristics of each tracked trajectory, each tracked trajectory can be used as a corresponding vertex and connected to construct a corresponding trajectory graph. The vertex characteristics of each vertex in the trajectory graph are determined according to the attribute characteristics of each tracked trajectory.

[0089] In addition, when constructing the target map of the target to be tracked in the current video frame, the current video frame can also be input into the target map extraction network in the pre-trained multi-target tracking model, and the target map extraction network is used to construct the target map of the target to be tracked in the current video frame.

[0090] S420: Construct a target map according to the target to be tracked in the current video frame.

[0091] S430 , calculating vertex similarity and edge similarity between the target graph and the trajectory graph based on vertex features and edge features in the target graph and the trajectory graph.

[0092] Optionally, after constructing the target graph of the target to be tracked and the trajectory graph of the tracked trajectory, the vertex features of each vertex in the target graph and the trajectory graph and the edge features of each edge are input into the graph matching network in the trained multi-target tracking model. The vertex similarity conversion layer in the graph matching network is used to analyze the feature similarity between each vertex in the target graph and each vertex in the trajectory graph to calculate the vertex similarity between the target graph and the trajectory graph. At the same time, the edge similarity conversion layer in the graph matching network is used to analyze the feature similarity between each edge in the target graph and each edge in the trajectory graph to calculate the edge similarity between the target graph and the trajectory graph. At this time, the vertex similarity and edge similarity between the target graph and the trajectory graph can accurately describe the similarity between every two vertices in the target graph and the trajectory graph and the similarity between every two edges, so that the degree of matching between each target to be tracked in the target graph and each tracked trajectory in the trajectory graph can be comprehensively analyzed by analyzing the vertex similarity and edge similarity between the target graph and the trajectory graph.

[0093] In addition, in order to further improve the matching accuracy between each vertex in the target graph and the trajectory graph, and to make it easier to distinguish unmatched vertices through the vertex features of each vertex in the target graph and the trajectory graph, this embodiment also performs feature enhancement on the vertex features of each vertex in the target graph and the trajectory graph and the edge features of each edge through the feature enhancement network in the multi-target tracking model; specifically, after constructing the target graph and the trajectory graph, as shown in FIG. Figure 4B As shown, the vertex features and edge features in the target graph and trajectory graph are input into the pre-built feature enhancement network to enhance the vertex features and edge features in the target graph and trajectory graph; the enhanced vertex features and edge features in the target graph and trajectory graph are used to calculate the vertex similarity and edge similarity between the target graph and the trajectory graph.

[0094] Specifically, according to the feature enhancement method proposed for the feature enhancement network in the training method of the multi-target tracking model provided in the above embodiment, the feature similarity between each vertex in the target graph and each vertex in the trajectory graph is used as the corresponding weight, and then the vertex features of each vertex in the trajectory graph are weightedly fused into the vertex features of each vertex in the target graph, and the vertex features of each vertex in the target graph are weightedly fused into the vertex features of each vertex in the trajectory graph, thereby enhancing the vertex features of each vertex in the target graph and the trajectory graph. The enhanced vertex features are concatenated to obtain enhanced edge features; or according to the above feature enhancement method, the feature similarity between each edge in the target graph and each edge in the trajectory graph is used as the corresponding weight, and then the edge features of each edge in the trajectory graph are weightedly fused into the edge features of each edge in the target graph, and the edge features of each edge in the target graph are weightedly fused into the edge features of each edge in the trajectory graph, thereby enhancing the edge features of each edge in the target graph and the trajectory graph, and subsequently using the enhanced vertex features and edge features to calculate the vertex similarity and edge similarity between the target graph and the trajectory graph.

[0095] S440 , calculating a matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph according to the vertex similarity and the edge similarity.

[0096] Optionally, after calculating the vertex similarity and edge similarity between the target graph and the trajectory graph, since the number of vertices and the number of edges in the target graph and the trajectory graph may be different, the dimensions of the vertex similarity and edge similarity between the target graph and the trajectory graph may be different. In order to ensure the efficiency and accuracy of calculating the corresponding matching score by performing a fusion analysis on the vertex similarity and the edge similarity, this embodiment will also set corresponding conversion dimensions for the vertex similarity and the edge similarity according to the number of vertices in the target graph and the trajectory graph, so as to perform a dimension conversion on the vertex similarity preliminarily calculated by the vertex similarity conversion layer in the graph matching network, and perform a dimension conversion on the edge similarity preliminarily calculated by the edge similarity conversion layer in the graph matching network, so that the vertex similarity and edge similarity after the dimension conversion can be accurately and efficiently integrated. Then, through the quadratic programming layer in the graph matching network, the vertex similarity and edge similarity after the dimension conversion are directly substituted into the pre-relaxed quadratic programming problem, thereby calculating the matching score between each target to be tracked in the target graph and each tracked trajectory in the trajectory graph.

[0097] For example, Figure 4BAs shown in the figure, if there are four targets to be tracked in the current video frame and five tracked tracks in the historical video frame, the constructed target graph contains four vertices and six edges, and the track graph contains five vertices and ten edges. The target graph and track graph are then fed into a feature enhancement network. After enhancing the vertex and edge features in both the target and track graphs, vertex and edge similarities between the target and track graphs can be calculated based on the enhanced vertex and edge features. Assuming an edge consists of two vertices, A and B, the edge is calculated as two edges: an AB edge from A to B and a BA edge from B to A. The corresponding edge features for this edge include the concatenation of the vertex features from A to B and the concatenation of the vertex features from B to A. The corresponding edge similarity also includes the similarity between edges in the same direction. In this case, vertex similarity is a 4*5 matrix, and edge similarity is a 12*20 matrix. At this time, in order to ensure that vertex similarity and edge similarity can be accurately and efficiently integrated, vertex similarity will be dimensionally converted through the vertex similarity conversion layer, and edge similarity will be dimensionally converted through the edge similarity conversion layer. For example, vertex similarity is directly stretched to a 1*20 matrix, while edge similarity is similarity expanded and padded and converted to a 20*20 matrix, so that the vertex similarity and edge similarity after dimension conversion can be accurately and efficiently integrated. At this time, through the quadratic programming layer, the vertex similarity and edge similarity after dimension conversion are directly substituted into the pre-relaxed quadratic programming problem. The calculated matching score between each target to be tracked and each tracked trajectory in the trajectory graph is a 4*5 matrix to represent the degree of matching between each vertex in the target graph and each vertex in the trajectory graph, thereby facilitating the subsequent determination of the matching trajectory of each target to be tracked according to the matching score.

[0098] S450: Determine a matching trajectory of the target to be tracked according to the matching score.

[0099] The technical solution provided in this embodiment utilizes a trajectory graph extraction network within a pre-trained multi-target tracking model to construct a trajectory graph of tracked trajectories in historical video frames, and uses a target graph extraction network within the multi-target tracking model to construct a target graph of the target to be tracked in the current video frame. Then, a graph matching network comprehensively refers to the vertex features and edge features in the target graph and trajectory graph to calculate the matching score between each target to be tracked and each tracked trajectory in the trajectory graph, thereby determining the matching trajectory of each target to be tracked, achieving accurate tracking of multiple targets in the video, preventing tracking failure when a target is occluded, avoiding the limitations of video multi-target tracking, and improving the robustness of video multi-target tracking.

[0100] Example 5

[0101] Figure 5This is a flowchart of a multi-target tracking method provided by the fifth embodiment of the present invention. This embodiment is optimized based on the above embodiment. Specifically, Figure 5 As shown, this embodiment mainly explains in detail the specific matching process of determining the matching trajectory of each target to be tracked from the tracked trajectories according to the matching score.

[0102] Optional, such as Figure 5 As shown, this embodiment may include the following steps:

[0103] S510: Construct a target map according to the target to be tracked in the current video frame.

[0104] S520 , performing graph matching on the target graph and the existing trajectory graph to calculate a matching score between the target to be tracked and the tracked trajectory in the trajectory graph.

[0105] S530 : For each target to be tracked, select the highest matching score from the matching scores between the target to be tracked and each tracked track.

[0106] Optionally, after calculating the matching score between each target to be tracked in the target map and each tracked track in the track map, a specific matching score is assigned to each target to be tracked and each tracked track. Therefore, when searching for the tracked track that best matches the target to be tracked, the first step is to filter out the highest matching score from the matching scores between the target to be tracked and each tracked track. The tracked track with the highest matching score can then be considered the track that best matches the target to be tracked among all the tracked tracks. Subsequently, a further determination is made as to whether the tracked track with the highest matching score is the true matching track for the track to be tracked.

[0107] S540 : Determine a matching trajectory of each target to be tracked according to the highest matching score of each target to be tracked and a preset intra-frame target addition condition.

[0108] Optionally, since a target to be tracked in the current video frame may be newly added and has not appeared in the historical video frames, there is no matching trajectory of the newly added target to be tracked in the tracked trajectories under the historical video frames. Therefore, when predicting the matching trajectory of each track to be tracked, it is first necessary to determine whether each target to be tracked is a newly added target in the current video frame. In view of the characteristics of the newly added target, this embodiment pre-sets an intra-frame target addition condition, and the intra-frame target addition condition can specifically include at least one of the following conditions: 1) the vertex similarity between the target to be tracked and each tracked trajectory in the trajectory map is less than a preset similarity threshold; 2) the position difference between the center position of the target to be tracked in the current video frame and the target center position in each tracked trajectory in the trajectory map is greater than a preset position difference; 3) there is no intersection between the target detection box of the target to be tracked in the current video frame and the target detection box of each tracked trajectory in the trajectory map.

[0109] For point 1), if the vertex similarity between a target to be tracked and each tracked track in the trajectory map is less than a preset similarity threshold, it indicates that the target to be tracked has a low degree of match with each tracked track in the trajectory map. Therefore, the target to be tracked is added to the current video frame. For point 2), the target center coordinates of the tracked track can be obtained by averaging the center coordinates of the associated targets pointed to by each tracked track in each historical video frame. If the position difference between the center coordinates of the target to be tracked in the current video frame and the center coordinates of the targets in each tracked track in the trajectory map is greater than a preset position difference, it indicates that the position of the target to be tracked is significantly different from that of each tracked track. Therefore, the target to be tracked is added to the current video frame. For point 3), when identifying each target to be tracked in the current video frame, a target detection frame is marked for each target to be tracked. Therefore, target detection frames are also present for the associated targets of the tracked track in the historical video frames. By averaging the target detection frames of the associated targets, the target detection frames of each tracked track can also be obtained. At this time, if there is no intersection between the target detection frame of a certain target to be tracked in the current video frame and the target detection frame of each tracked track in the track map, it means that the position of the target to be tracked is far away from that of each tracked track. Therefore, the target to be tracked is added as a new target in the current video frame.

[0110] At this time, after filtering out the highest matching score between each target to be tracked and the tracked trajectory, it is first determined whether each target to be tracked meets the preset in-frame target addition conditions. If a target to be tracked meets at least one of the above-mentioned in-frame target addition conditions, it can be determined that the target to be tracked is a newly added target in the current video frame, indicating that the target to be tracked has never appeared in the historical video frame, and the trajectory of the target to be tracked starts from the current video frame. Therefore, the position of the newly added target in the current video frame can be directly used as the matching trajectory of the newly added target. However, for each remaining target to be tracked in the current video frame except the newly added target, the remaining target to be tracked has already appeared in the historical video frame, that is, the trajectory of the remaining target to be tracked exists in each tracked trajectory under the historical video frame. Therefore, the tracked trajectory pointed to by the highest matching score of the remaining target to be tracked can be directly found from each tracked trajectory under the historical video frame, and the found tracked trajectory can be used as the matching trajectory of the remaining target to be tracked.

[0111] In addition, in order to ensure the accuracy of multi-target tracking, this embodiment determines the matching trajectory of each target to be tracked according to the highest matching score of each target to be tracked and the preset intra-frame target addition condition. For each target to be tracked, the highest matching score among the matching scores between the target to be tracked and the tracked trajectories will be rounded up, and the other matching scores will be rounded down, so as to greedily round the matching scores between the target to be tracked in the target map and the tracked trajectories in the trajectory map to form the corresponding optimal matching score. The optimal matching score is a 0-1 matrix after the ordinary score matrix is converted. Subsequently, the matching trajectory of each target to be tracked can be determined directly according to the optimal matching score and the preset intra-frame target addition condition according to the above steps.

[0112] The technical solution provided by this embodiment, after calculating the matching scores between the targets to be tracked in the target map and the tracked trajectories in the trajectory map, can screen the highest matching score for each target to be tracked from the matching scores between the target to be tracked and each tracked trajectories. Then, according to the highest matching score of each target to be tracked and the preset intra-frame target addition condition, the matching trajectory of each target to be tracked is determined, thereby screening out the newly added targets in the current video frame, achieving accurate tracking of multiple targets in the video, and improving the robustness of multi-target tracking in the video.

[0113] Example 6

[0114] Figure 6 A structural diagram of a multi-target tracking model training device provided in Example 6 of the present invention is shown in FIG. Figure 6 As shown, the device may include:

[0115] A model construction module 610 is configured to construct an initial multi-target tracking model, the multi-target tracking model comprising a target graph extraction network, a trajectory graph extraction network, and a graph matching network; the target graph extraction network is configured to construct a target graph based on the target to be tracked in the current video frame, the trajectory graph extraction network is configured to construct a trajectory graph based on the tracked trajectories in the historical video frames, and the graph matching network is configured to perform graph matching on the target graph and the trajectory graph to predict the matching trajectory of the target to be tracked; wherein both the target graph and the trajectory graph comprise vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked, the vertices of the trajectory graph correspond to the tracked trajectories, and the edge features of the edges between two vertices in the target graph and the trajectory graph comprise attribute relationships between the two vertices;

[0116] The model training module 620 is used to use the preset loss function of the multi-target tracking model to jointly train the various networks in the multi-target tracking model based on the actual matching trajectory of the target to be tracked in the training sample and the predicted matching trajectory to obtain a trained multi-target tracking model.

[0117] The technical solution provided by this embodiment is that if multi-target tracking is achieved through a multi-target tracking model, the multi-target tracking model needs to be trained first. At this time, the initially constructed multi-target tracking model includes a target graph extraction network, a trajectory graph extraction network and a graph matching network, which are respectively used to perform the target graph construction, trajectory graph construction and graph matching steps in the multi-target tracking process, and set a corresponding preset loss function for the multi-target tracking model. Then, using the preset loss function, according to the actual matching trajectory of each target to be tracked in the training sample and the matching trajectory predicted by the multi-target tracking model, the various networks in the multi-target tracking model are jointly trained to obtain the trained multi-target tracking model. There is no need to specifically train each network in the multi-target tracking model separately, thereby improving the correlation between the networks in the multi-target tracking model and the accuracy of multi-target tracking.

[0118] Furthermore, the above-mentioned vertices have vertex features, the vertex features of the vertices in the target graph are the attribute features of the target to be tracked corresponding to the vertex, the vertex features of the vertices in the trajectory graph are the attribute features of the tracked trajectory corresponding to the vertex, and the attribute relationship between the two vertices in the edge features of the edges between the two vertices in the target graph and the trajectory graph is the relationship between the attribute features of the two vertices.

[0119] Furthermore, the above-mentioned multi-target tracking model can also include a feature enhancement network, which is used to enhance the vertex features and edge features in the target graph and the trajectory graph, so that the graph matching network can perform graph matching on the target graph and the trajectory graph based on the enhanced vertex features and edge features in the target graph and the trajectory graph.

[0120] Furthermore, the above-mentioned graph matching network can be specifically used to calculate the vertex similarity and edge similarity between the target graph and the trajectory graph based on the enhanced vertex features and edge features in the target graph and the trajectory graph, calculate the matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph based on the vertex similarity and edge similarity, and predict the matching trajectory of the target to be tracked based on the matching score.

[0121] Furthermore, the training samples may include real matching trajectories of the target to be tracked in each current video frame in historical video frames. The model training module 620 may be specifically used to:

[0122] The target map extraction network in the multi-target tracking model is used to construct the target map of the target to be tracked in the current video frame in the training sample;

[0123] The trajectory graph extraction network in the multi-target tracking model is used to construct a trajectory graph of the tracked trajectories in the historical video frames in the training sample;

[0124] The feature enhancement network in the multi-target tracking model is used to enhance the vertex features and edge features in the target graph and trajectory graph. The enhanced vertex features and edge features in the target graph and trajectory graph are then input into the graph matching network in the multi-target tracking model to predict the matching trajectory of the target to be tracked.

[0125] Utilizing the preset loss function of the multi-target tracking network, the gradients of the graph matching network, feature enhancement network, trajectory graph extraction network, and target graph extraction network are back-propagated according to the actual matching trajectory and predicted matching trajectory of each target to be tracked in the current video frame in the training sample in the historical video frame, so as to jointly train the networks in the multi-target tracking model.

[0126] Furthermore, the above-mentioned preset loss function can be a weighted binary cross entropy loss function, and the weights in the preset loss function are set according to the number of vertices in the target graph and the trajectory graph; the feature enhancement network is a cross-graph convolutional network.

[0127] Furthermore, the above-mentioned graph matching network can be composed of a vertex similarity conversion layer, an edge similarity conversion layer and a quadratic programming layer. The vertex similarity conversion layer is used to calculate the vertex similarity between the target graph and the trajectory graph and perform dimensionality conversion. The edge similarity conversion layer is used to calculate the edge similarity between the target graph and the trajectory graph and perform dimensionality conversion. The quadratic programming layer is used to calculate the matching score based on the quadratic programming problem using the vertex similarity and edge similarity after dimensionality conversion.

[0128] Furthermore, the above quadratic programming problem can be obtained by adjusting the discrete allocation objective in the quadratic assignment problem to the corresponding continuous programming objective and then performing corresponding conversion on the quadratic assignment problem.

[0129] Furthermore, the vertex similarity conversion layer and the edge similarity conversion layer can set corresponding conversion dimensions according to the number of vertices in the target graph and the trajectory graph.

[0130] Furthermore, the attribute features of the above vertices can be expressed as vector features, and the edge features of the edge between two vertices can be expressed as a concatenated vector or a weighted vector of the vector features corresponding to the two vertices.

[0131] The training device for the multi-target tracking model provided in this embodiment can be applied to the training method for the multi-target tracking model provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0132] Example 7

[0133] Figure 7 A schematic diagram of the structure of a multi-target tracking device provided in Example 7 of the present invention is shown in FIG. Figure 7 As shown, the device may include:

[0134] A target graph construction module 710 is configured to construct a target graph based on the target to be tracked in the current video frame; wherein the target graph includes vertices and edges connecting the vertices, wherein the vertices of the target graph correspond to the target to be tracked, and the edge features of the edges between two vertices include the attribute relationship between the two vertices;

[0135] A graph matching module 720 is configured to perform graph matching on a target graph and an existing trajectory graph to calculate a matching score between the target to be tracked and the tracked track in the trajectory graph; wherein the trajectory graph includes vertices and edges connecting the vertices, the vertices of the trajectory graph correspond to the existing tracked tracks, and the edge features of the edge between two vertices include an attribute relationship between the two vertices;

[0136] The trajectory determination module 730 is configured to determine a matching trajectory of the target to be tracked according to the matching score.

[0137] The technical solution provided by this embodiment, when tracking the trajectory of each target to be tracked in the current video frame, has determined each tracked trajectory for the historical video frame and constructed a corresponding trajectory graph, each vertex of the trajectory graph corresponds to each existing tracked trajectory, and the edge feature of the edge between two vertices includes the attribute relationship between the two vertices. At this time, a target graph is constructed according to each target to be tracked in the current video frame, each vertex of the target graph corresponds to each target to be tracked, and the edge feature of the edge between two vertices includes the attribute relationship between the two vertices, and then the target graph and the trajectory graph are matched to calculate the matching score between each target to be tracked and each tracked trajectory in the trajectory graph, so as to determine the matching trajectory of each target to be tracked, and realize accurate tracking of multiple targets in the video. When matching the graph, the edge features in the target graph and the trajectory graph are fully referenced to prevent the problem of tracking failure when a target is blocked, avoid the limitations of video multi-target tracking, and improve the robustness of video multi-target tracking.

[0138] Furthermore, the graph matching module 720 may include:

[0139] A similarity calculation unit, configured to calculate vertex similarity and edge similarity between the target graph and the trajectory graph based on vertex features and edge features in the target graph and the trajectory graph;

[0140] The matching score calculation unit is used to calculate the matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph based on the vertex similarity and the edge similarity.

[0141] Furthermore, the above-mentioned similarity calculation unit can be specifically used to: input the vertex features and edge features in the target graph and trajectory graph into a pre-built feature enhancement network, and perform feature enhancement on the vertex features and edge features in the target graph and trajectory graph; use the enhanced vertex features and edge features in the target graph and trajectory graph to calculate the vertex similarity and edge similarity between the target graph and the trajectory graph.

[0142] Furthermore, the multi-target tracking device may further include: a trajectory map construction module, configured to construct a trajectory map based on tracked trajectories in historical video frames.

[0143] Furthermore, the trajectory graph construction module may include:

[0144] an associated target determining unit, configured to determine, for each tracked trajectory, an associated target of the tracked trajectory in each historical video frame;

[0145] an attribute feature calculation unit, configured to extract attribute sub-features of the tracked trajectory associated with the target in each historical video frame, and calculate a mean of the corresponding attribute sub-features as the attribute feature of the tracked trajectory;

[0146] The trajectory graph construction unit is used to construct a trajectory graph with each tracked trajectory as a vertex, and obtain a vertex feature of each vertex of the trajectory graph.

[0147] Furthermore, the attribute feature calculation unit may be specifically configured to calculate a corresponding attribute sub-feature mean value based on the attribute sub-features of the target associated with the tracked trajectory in each historical video frame by using a sliding average method.

[0148] Furthermore, the trajectory determination module 730 may include:

[0149] a matching score screening unit, configured to screen, for each target to be tracked, the highest matching score from the matching scores between the target to be tracked and each tracked trajectory;

[0150] The track determination unit is used to determine the matching track of each target to be tracked according to the highest matching score of each target to be tracked and the preset intra-frame target addition condition.

[0151] Furthermore, the trajectory determination unit may be specifically configured to: for each target to be tracked, round up the highest matching score among the matching scores between the target to be tracked and each tracked trajectory, and round down the other matching scores to form a corresponding optimal matching score; and determine the matching trajectory of each target to be tracked according to the optimal matching score and a preset intra-frame target addition condition.

[0152] Furthermore, the trajectory determination unit may be specifically configured to: determine, among the targets to be tracked in the current video frame, those that meet the conditions for adding a new target within the frame as newly added targets, and use the position of the newly added target in the current video frame as the matching trajectory of the newly added target; and for each remaining target to be tracked in the current video frame other than the newly added target, use the tracked trajectory pointed to by the highest matching score of the remaining target to be tracked as the matching trajectory of the remaining target to be tracked.

[0153] Furthermore, the above-mentioned new conditions for in-frame targets may include at least one of the following conditions: the vertex similarity between the target to be tracked and each tracked trajectory in the trajectory map is less than a preset similarity threshold; the position difference between the center position of the target to be tracked in the current video frame and the target center position in each tracked trajectory in the trajectory map is greater than a preset position difference; there is no intersection between the target detection box of the target to be tracked in the current video frame and the target detection box of each tracked trajectory in the trajectory map.

[0154] Furthermore, the target graph construction module 710 may be specifically configured to: extract attribute features of each target to be tracked in the current video frame; construct a target graph with each target to be tracked as a vertex, and obtain vertex features of each vertex of the target graph.

[0155] Furthermore, the above-mentioned multi-target tracking device can also include: a trajectory map updating module, which is used to update the trajectory map according to the matching trajectory of the target to be tracked in the current video frame, and update the vertex features of each vertex in the trajectory map according to the attribute features of the matching trajectory of each target to be tracked in the current video frame.

[0156] The multi-target tracking device provided in this embodiment can be applied to the multi-target tracking method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0157] Example 8

[0158] Figure 8 This is a schematic diagram of the structure of a computing device provided by Example 8 of the present invention. Figure 8 As shown, the computing device includes a processor 810, a storage device 820 and a communication device 830; the number of processors 810 in the computing device can be one or more. Figure 8 In the figure, a processor 810 is used as an example; the processor 810, the storage device 820 and the communication device 830 of the computing device can be connected via a bus or other means. Figure 8 The bus connection is taken as an example.

[0159] The storage device 820 is a computer-readable storage medium that can be used to store software programs, computer executable programs, and modules. The processor 810 executes the software programs, instructions, and modules stored in the storage device 820 to perform various functional applications and data processing of the computing device, that is, to implement the above-mentioned multi-target tracking model training method or multi-target tracking method.

[0160] The storage device 820 may primarily include a program storage area and a data storage area. The program storage area may store an operating system and at least one application required for a function; the data storage area may store data generated based on the terminal's usage. Furthermore, the storage device 820 may include high-speed random access memory and non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state memory device. In some examples, the storage device 820 may further include memory remotely located from the multi-function controller, which may be connected to the vehicle via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0161] The communication device 830 can be used to implement a network connection or a mobile data connection between computing devices.

[0162] A computing device provided in this embodiment can be used to execute the training method of the multi-target tracking model or the multi-target tracking method provided in any of the above embodiments, and has corresponding functions and beneficial effects.

[0163] Embodiment 9

[0164] Embodiment 9 of the present invention further provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the program can implement the multi-target tracking model training method or multi-target tracking method described in any of the above embodiments. Of course, the computer-executable instructions in a storage medium containing computer-executable instructions provided in an embodiment of the present invention are not limited to the operations described above, and can also perform related operations in the multi-target tracking model training method or multi-target tracking method described in any embodiment of the present invention.

[0165] Through the above description of the implementation methods, those skilled in the art can clearly understand that the present invention can be implemented with the help of software and necessary general-purpose hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of the present invention is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0166] It is worth noting that in the embodiments of the training device of the above-mentioned multi-target tracking model or the multi-target tracking device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific names of the functional units are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of the present invention.

[0167] Note that the above are only preferred embodiments of the present invention and the technical principles employed. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments, and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in detail through the above embodiments, the present invention is not limited to the above embodiments and may include many other equivalent embodiments without departing from the concept of the present invention. The scope of the present invention is determined by the scope of the appended claims.

Claims

1. A training method for a multi-target tracking model, characterized in that: include: Constructing an initial multi-target tracking model, wherein the multi-target tracking model includes a target graph extraction network, a trajectory graph extraction network, and a graph matching network; The target graph extraction network is used to construct a target graph based on the target to be tracked in the current video frame, the trajectory graph extraction network is used to construct a trajectory graph based on the tracked trajectories in the historical video frames, and the graph matching network is used to perform graph matching on the target graph and the trajectory graph to predict the matching trajectory of the target to be tracked; wherein, the target graph and the trajectory graph both include vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked, the vertices of the trajectory graph correspond to the tracked trajectories, and the edge features of the edges between two vertices in the target graph and the trajectory graph include the attribute relationship between the two vertices; Using a preset loss function of the multi-target tracking model, and based on the actual matching trajectories and predicted matching trajectories of the targets to be tracked in the training samples, the networks in the multi-target tracking model are jointly trained to obtain a trained multi-target tracking model; in, The trajectory graph extraction network is further used to: identify the tracked trajectories formed by the same target in each historical video frame, and obtain the attribute features of the tracked trajectory corresponding to the target by performing feature fusion on the attribute features of the same target in each historical video frame, wherein the vertex features of the vertices in the trajectory graph are the attribute features of the tracked trajectory corresponding to the vertex, and the vertex features of the vertices in the target graph are the attribute features of the target to be tracked corresponding to the vertex; The graph matching network is further used to perform graph matching on the target graph and the trajectory graph according to vertex features of each vertex and edge features of each edge in the target graph and the trajectory graph.

2. The method according to claim 1, characterized in that The attribute relationship between the two vertices in the edge features of the edges of the two vertices in the target graph and the trajectory graph is the relationship between the attribute features of the two vertices.

3. The method according to claim 2, characterized in that The multi-target tracking model further includes a feature enhancement network, which is used to enhance vertex features and edge features in the target graph and the trajectory graph, so that the graph matching network performs graph matching on the target graph and the trajectory graph according to the enhanced vertex features and edge features in the target graph and the trajectory graph; For each vertex feature in the target graph: the feature enhancement network is used to analyze the feature similarity between the vertex and each vertex in the trajectory graph as the weight between the vertex and each vertex in the trajectory graph, and use the weight between the vertex and each vertex in the trajectory graph to weightedly fuse the vertex features of each vertex in the trajectory graph into the vertex feature of the vertex, thereby obtaining the vertex feature of the vertex after feature enhancement; For the vertex features of each vertex in the trajectory graph: the feature enhancement network is used to analyze the feature similarity between the vertex and each vertex in the target graph as the weight between the vertex and each vertex in the target graph, and use the weights between the vertex and each vertex in the target graph to weightedly fuse the vertex features of each vertex in the target graph into the vertex features of the vertex, thereby obtaining the vertex features of the vertex in the trajectory graph after feature enhancement.

4. The method according to claim 3, characterized in that The graph matching network is specifically used to calculate the vertex similarity and edge similarity between the target graph and the trajectory graph based on the enhanced vertex features and edge features in the target graph and the trajectory graph, calculate the matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph based on the vertex similarity and the edge similarity, and predict the matching trajectory of the target to be tracked based on the matching score.

5. The method according to claim 4, characterized in that The training sample includes the real matching trajectory of the target to be tracked in each current video frame in the historical video frame. The preset loss function of the multi-target tracking model is used to jointly train each network in the multi-target tracking model according to the real matching trajectory and the predicted matching trajectory of the target to be tracked in the training sample, including: Constructing a target map of the target to be tracked in the current training frame in the training sample using a target map extraction network in the multi-target tracking model; Constructing a trajectory map of tracked trajectories in historical training frames within the training sample using a trajectory map extraction network within the multi-target tracking model; Performing feature enhancement on vertex features and edge features in the target graph and the trajectory graph using a feature enhancement network within the multi-target tracking model, and inputting the enhanced vertex features and edge features in the target graph and the trajectory graph into a graph matching network within the multi-target tracking model to predict matching trajectories of the target to be tracked; Utilizing the preset loss function of the multi-target tracking model, the gradient backpropagation of the graph matching network, feature enhancement network, trajectory graph extraction network and target graph extraction network is performed according to the actual matching trajectory and predicted matching trajectory of each target to be tracked in the current video frame in the training sample in the historical video frame, so as to jointly train the networks in the multi-target tracking model.

6. The method according to claim 5, characterized in that The preset loss function is a weighted binary cross entropy loss function, and the weights in the preset loss function are set according to the number of vertices in the target graph and the trajectory graph; the feature enhancement network is a cross-graph convolutional network.

7. The method according to claim 4, characterized in that The graph matching network consists of a vertex similarity conversion layer, an edge similarity conversion layer, and a quadratic programming layer. The vertex similarity conversion layer is used to calculate the vertex similarity between the target graph and the trajectory graph and perform dimensionality conversion. The edge similarity conversion layer is used to calculate the edge similarity between the target graph and the trajectory graph and perform dimensionality conversion. The quadratic programming layer is used to calculate the matching score based on the quadratic programming problem using the vertex similarity and edge similarity after dimensionality conversion.

8. The method according to claim 7, characterized in that The quadratic programming problem is obtained by adjusting the discrete allocation objective in the quadratic assignment problem to a corresponding continuous programming objective and then performing corresponding conversion on the quadratic assignment problem.

9. The method according to claim 7, characterized in that The vertex similarity conversion layer and the edge similarity conversion layer set corresponding conversion dimensions according to the number of vertices in the target graph and the trajectory graph.

10. The method according to claim 1, characterized in that The attribute feature of the vertex is represented as a vector feature, and the edge feature of the edge between two vertices is represented as a concatenated vector or a weighted vector of the vector features corresponding to the two vertices.

11. A multi-target tracking method, characterized in that: include: Constructing a trajectory graph based on tracked trajectories in historical video frames specifically includes: identifying tracked trajectories formed by the same target in each historical video frame, and fusing attribute features of the same target in each historical video frame to obtain attribute features of the tracked trajectories corresponding to the target, and using the attribute features of the tracked trajectories as vertex features of vertices in the trajectory graph; Constructing a target graph based on the target to be tracked in the current video frame; wherein the target graph includes vertices and edges connecting the vertices, the vertices of the target graph correspond to the target to be tracked, and the edge features of the edges between two vertices include the attribute relationship between the two vertices, and the vertex features of the vertices in the target graph are the attribute features of the target to be tracked corresponding to the vertices; Performing graph matching on the target graph and the existing trajectory graph based on vertex features of each vertex and edge features of each edge in the target graph and the trajectory graph to calculate a matching score between the target to be tracked and the tracked track in the trajectory graph; wherein the trajectory graph includes vertices and edges connecting vertices, the vertices of the trajectory graph correspond to the existing tracked tracks, and the edge features of the edge between two vertices include an attribute relationship between the two vertices; A matching trajectory of the target to be tracked is determined according to the matching score.

12. The method according to claim 11, characterized in that The vertex has a vertex feature, the vertex feature of the vertex in the target graph is the attribute feature of the target to be tracked corresponding to the vertex, the vertex feature of the vertex in the trajectory graph is the attribute feature of the tracked trajectory corresponding to the vertex, and the attribute relationship between two vertices in the target graph and the trajectory graph is the relationship between the attribute features of the two vertices.

13. The method according to claim 12, characterized in that The performing graph matching on the target graph and the existing trajectory graph to calculate a matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph includes: Calculating vertex similarity and edge similarity between the target graph and the trajectory graph based on vertex features and edge features in the target graph and the trajectory graph; A matching score between the target to be tracked in the target graph and the tracked trajectory in the trajectory graph is calculated according to the vertex similarity and the edge similarity.

14. The method according to claim 13, characterized in that The calculating the vertex similarity and edge similarity between the target graph and the trajectory graph according to the vertex features and edge features in the target graph and the trajectory graph includes: Inputting the vertex features and edge features in the target graph and the trajectory graph into a pre-built feature enhancement network, and performing feature enhancement on the vertex features and edge features in the target graph and the trajectory graph; Calculating vertex similarity and edge similarity between the target graph and the trajectory graph using the enhanced vertex features and edge features in the target graph and the trajectory graph; For each vertex feature in the target graph: the feature enhancement network is used to analyze the feature similarity between the vertex and each vertex in the trajectory graph as the weight between the vertex and each vertex in the trajectory graph, and use the weight between the vertex and each vertex in the trajectory graph to weightedly fuse the vertex features of each vertex in the trajectory graph into the vertex feature of the vertex, thereby obtaining the vertex feature of the vertex after feature enhancement; For the vertex features of each vertex in the trajectory graph: the feature enhancement network is used to analyze the feature similarity between the vertex and each vertex in the target graph as the weight between the vertex and each vertex in the target graph, and use the weights between the vertex and each vertex in the target graph to weightedly fuse the vertex features of each vertex in the target graph into the vertex features of the vertex, thereby obtaining the vertex features of the vertex in the trajectory graph after feature enhancement.

15. The method according to claim 14, characterized in that The constructing the trajectory map according to the tracked trajectories in the historical video frames includes: For each tracked trajectory, determining an associated target of the tracked trajectory in each historical video frame; Extract the attribute sub-features of the tracked trajectory associated with the target in each historical video frame, and calculate the corresponding attribute sub-feature mean as the attribute feature of the tracked trajectory; The trajectory graph is constructed with each tracked trajectory as a vertex, and a vertex feature of each vertex of the trajectory graph is obtained.

16. The method according to claim 15, characterized in that The calculation of the corresponding attribute sub-feature mean includes: The sliding average method is used to calculate the corresponding attribute sub-feature mean based on the attribute sub-features of the target associated with the tracked trajectory in each historical video frame.

17. The method according to claim 11, characterized in that Determining the matching trajectory of the target to be tracked according to the matching score includes: For each target to be tracked, select the highest matching score from the matching scores between the target to be tracked and each tracked trajectory; According to the highest matching score of each target to be tracked and the preset intra-frame target addition condition, the matching trajectory of each target to be tracked is determined.

18. The method according to claim 17, characterized in that The step of determining the matching trajectory of each target to be tracked according to the highest matching score of each target to be tracked and the preset intra-frame target addition condition includes: For each target to be tracked, the highest matching score among the matching scores between the target to be tracked and each tracked trajectory is rounded up, and the other matching scores are rounded down to form the corresponding optimal matching score; According to the optimal matching score and the preset intra-frame target addition condition, a matching trajectory of each target to be tracked is determined.

19. The method according to claim 17, wherein The step of determining the matching trajectory of each target to be tracked according to the highest matching score of each target to be tracked and the preset intra-frame target addition condition includes: A target to be tracked in the current video frame that meets the in-frame target addition condition is determined as a newly added target, and a position of the newly added target in the current video frame is used as a matching trajectory of the newly added target.

20. The method according to claim 19, characterized in that The step of determining the matching trajectory of each target to be tracked according to the highest matching score of each target to be tracked and the preset intra-frame target addition condition further includes: For each remaining target to be tracked in the current video frame except the newly added target, the tracked trajectory pointed to by the highest matching score of the remaining target to be tracked is used as the matching trajectory of the remaining target to be tracked.

21. The method according to claim 17, wherein The intra-frame target newly added condition includes at least one of the following conditions: The vertex similarity between the target to be tracked and each tracked trajectory in the trajectory graph is less than a preset similarity threshold; The position difference between the center position of the target to be tracked in the current video frame and the center position of the target in each tracked track in the track map is greater than a preset position difference value; There is no intersection between the target detection frame of the target to be tracked in the current video frame and the target detection frame of each tracked trajectory in the trajectory map.

22. The method according to any one of claims 11 to 21, characterized in that The step of constructing a target map according to the target to be tracked in the current video frame includes: Extracting attribute features of each target to be tracked in the current video frame; The target graph is constructed with each target to be tracked as a vertex, and a vertex feature of each vertex of the target graph is obtained.

23. The method according to any one of claims 11 to 21, characterized in that The target graph and the trajectory graph are complete graphs.

24. The method according to any one of claims 11 to 21, characterized in that After determining the matching trajectory of the target to be tracked according to the matching score, the method further includes: The trajectory map is updated according to the matching trajectory of the target to be tracked in the current video frame, and the vertex feature of each vertex in the trajectory map is updated according to the attribute feature of the matching trajectory of each target to be tracked in the current video frame.

25. A training device for a multi-target tracking model, characterized in that: include: A model building module is used to build an initial multi-target tracking model, wherein the multi-target tracking model includes a target graph extraction network, a trajectory graph extraction network and a graph matching network; The target graph extraction network is used to construct a target graph based on the target to be tracked in the current video frame, the trajectory graph extraction network is used to construct a trajectory graph based on the tracked trajectories in the historical video frames, and the graph matching network is used to perform graph matching on the target graph and the trajectory graph to predict the matching trajectory of the target to be tracked; wherein, the target graph and the trajectory graph both include vertices and edges connecting the vertices, the vertices of the target graph correspond to the targets to be tracked, the vertices of the trajectory graph correspond to the tracked trajectories, and the edge features of the edges between two vertices in the target graph and the trajectory graph include the attribute relationship between the two vertices; A model training module is used to jointly train the various networks in the multi-target tracking model based on the actual matching trajectories and predicted matching trajectories of the targets to be tracked in the training samples using the preset loss function of the multi-target tracking model to obtain a trained multi-target tracking model; in, The trajectory graph extraction network is further used to: identify the tracked trajectories formed by the same target in each historical video frame, and obtain the attribute features of the tracked trajectory corresponding to the target by performing feature fusion on the attribute features of the same target in each historical video frame, wherein the vertex features of the vertices in the trajectory graph are the attribute features of the tracked trajectory corresponding to the vertex, and the vertex features of the vertices in the target graph are the attribute features of the target to be tracked corresponding to the vertex; The graph matching network is further used to perform graph matching on the target graph and the trajectory graph according to vertex features of each vertex and edge features of each edge in the target graph and the trajectory graph.

26. A multi-target tracking device, characterized in that: include: The trajectory graph construction module is used to identify the tracked trajectories formed by the same target in each historical video frame, and obtain the attribute features of the tracked trajectory corresponding to the target by fusing the attribute features of the same target in each historical video frame, and use the attribute features of the tracked trajectory as the vertex features of the vertices in the trajectory graph; a target graph construction module, configured to construct a target graph based on the target to be tracked in the current video frame; wherein the target graph includes vertices and edges connecting the vertices, the vertices of the target graph correspond to the target to be tracked, and the edge features of the edges between two vertices include the attribute relationship between the two vertices, and the vertex features of the vertices in the target graph are the attribute features of the target to be tracked corresponding to the vertices; a graph matching module, configured to perform graph matching on the target graph and the existing trajectory graph based on vertex features of each vertex and edge features of each edge in the target graph and the trajectory graph, so as to calculate a matching score between the target to be tracked and the tracked track in the trajectory graph; wherein the trajectory graph includes vertices and edges connecting vertices, the vertices of the trajectory graph correspond to the existing tracked tracks, and the edge features of the edge between two vertices include an attribute relationship between the two vertices; The track determination module is used to determine the matching track of the target to be tracked according to the matching score.

27. A computing device, characterized in that The computing device comprises: one or more processors; a storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the training method of the multi-target tracking model as described in any one of claims 1-10, or implement the multi-target tracking method as described in any one of claims 11-24.

28. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, it implements the training method of the multi-target tracking model as described in any one of claims 1 to 10, or implements the multi-target tracking method as described in any one of claims 11 to 24.

Images