Multi-target tracking method, device, electronic device, storage medium and product

Abstract

The present invention provides a multi-target tracking method, device, electronic device, storage medium and product, the method comprising: obtaining a video to be tested; inputting the video to be tested into an end-to-end multi-target tracking model, and obtaining multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model; wherein the multi-target information includes a detection frame where the target is located and target identity information, the end-to-end multi-target model is obtained through training based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set. The present invention integrates target detection branches, feature branches and identity association branches through an end-to-end end-to-end multi-target tracking model, thereby solving the defect of inaccurate multi-target tracking results in the prior art and improving the accuracy of multi-target detection.

Description

Technical Field

[0001] The present invention relates to the field of target detection technology, and in particular to a multi-target tracking method, device, electronic equipment, storage medium and product. Background Art

[0002] Multi-target tracking aims to locate and identify all moving targets in a video, and has broad application prospects in visual surveillance, human-computer interaction, virtual reality, unmanned driving and other fields. Existing multi-target tracking methods can be divided into two frameworks: detection-based tracking framework and joint detection and feature tracking framework. As the mainstream algorithm for multi-target tracking, the detection-based tracking framework divides the multi-target tracking task into three independent modules: target detection, feature extraction, and identity association. Different from the detection-based tracking framework, the joint detection and feature tracking framework uses a one-stage tracker to simultaneously generate target detection and corresponding visual features. Relatively speaking, it can reduce the computational cost and build a real-time multi-target tracking algorithm.

[0003] However, the existing tracking framework faces the following problems: First, the identity association module is independent of the target detection and feature extraction modules, resulting in different modules not being able to gain mutual benefits. Second, since the errors generated by the previous module can be transmitted to the subsequent modules, executing multiple modules in sequence will lead to the accumulation of errors, which may eventually result in inaccurate detection results and tracking failure. Summary of the invention

[0004] The present invention provides a multi-target tracking method, device, electronic device, storage medium and product, which are used to solve the defect of inaccurate multi-target tracking results in the prior art and improve the multi-target detection accuracy.

[0005] The present invention provides a multi-target tracking method, comprising:

[0006] Get the video to be tested;

[0007] Inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0008] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0009] According to a multi-target tracking method provided by the present invention, the step of inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model comprises:

[0010] Extracting underlying features of a current frame of the video to be tested, and enhancing the underlying features according to historical trajectory features to obtain enhanced features of the current frame;

[0011] Obtaining a detection frame where the target is located according to the enhanced features of the current frame;

[0012] Obtaining a detection frame feature of the current frame according to the enhanced feature of the current frame and the detection frame where the target is located;

[0013] The target identity information of the current frame is obtained by performing identity association based on the detection frame where the target of the current frame is located and the detection frame features and the historical trajectory features.

[0014] According to a multi-target tracking method provided by the present invention, the step of enhancing the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame includes:

[0015] Obtaining the underlying features of the current frame and the historical trajectory features of the previous frame;

[0016] The continuous area between the underlying features of the current frame and the historical trajectory features of the previous frame is enhanced, and the interference area between the underlying features of the current frame and the historical trajectory features of the previous frame is erased to obtain the enhanced features of the current frame.

[0017] According to a multi-target tracking method provided by the present invention, obtaining the detection frame feature of the current frame according to the enhanced feature of the current frame and the detection frame where the target is located includes:

[0018] Global pooling is performed based on the enhanced features of the current frame and the detection frame where the target of the current frame is located to obtain the detection frame features of the current frame, wherein the detection frame where the target of the current frame is located has a one-to-one correspondence with the detection frame features of the current frame.

[0019] According to a multi-target tracking method provided by the present invention, the target identity information of the current frame is obtained by performing identity association based on the detection frame where the target of the current frame is located and the detection frame features and the historical trajectory features, including:

[0020] Generate a detection map by using the detection frame features of the current frame corresponding to the detection frame coordinates of the target in the current frame as nodes and the spliced ​​features of the detection frame features of the current frame as edges;

[0021] The trajectory features corresponding to the end coordinates of the historical trajectory of the previous frame are used as nodes, and the concatenated features of the trajectory features of the previous frame are used as edges to generate a trajectory graph;

[0022] Graph matching is performed based on the detection graph and the trajectory graph to obtain the target identity information of the current frame.

[0023] According to a multi-target tracking method provided by the present invention, the step of performing image matching based on the detection image and the trajectory image to obtain the target identity information of the current frame includes:

[0024] The detection image and the trajectory image are integrated with node features, a similarity matrix of the detection image and the trajectory image after the node feature integration is calculated, and target identity information of the current frame is obtained according to the similarity matrix.

[0025] The present invention also provides a multi-target tracking device, comprising:

[0026] An acquisition module is used to obtain the video to be tested;

[0027] An output module, used for inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0028] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0029] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of any of the multi-target tracking methods described above are implemented.

[0030] The present invention also provides a non-transitory computer-readable storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of any of the multi-target tracking methods described above are implemented.

[0031] The present invention also provides a computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps of any of the multi-target tracking methods described above are implemented.

[0032] The multi-target tracking method, device, electronic device, storage medium and product provided by the present invention integrate the target detection branch, feature branch and identity association branch through an end-to-end multi-target tracking model, and use trajectory information to enhance target detection and feature extraction to form positive feedback. In the present invention, false detection and missed detection are reduced by using trajectory information to enhance the underlying features of target detection. At the same time, the enhanced underlying features can assist the feature extraction model to reduce identity jumps. With more robust detection results and features, higher quality trajectories can be generated. BRIEF DESCRIPTION OF THE DRAWINGS

[0033] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0034] Figure 1 is a flow chart of a multi-target tracking method provided by an embodiment of the present invention;

[0035] Figure 2 is a schematic diagram of the structure of an end-to-end multi-target tracking model provided by an embodiment of the present invention;

[0036] Figure 3 is a schematic diagram of the structure of a feature enhancement module provided by an embodiment of the present invention;

[0037] Figure 4 is a schematic diagram of detection results of the multi-target tracking method provided by an embodiment of the present invention;

[0038] Figure 5 is a schematic diagram of the structure of a multi-target tracking device provided by an embodiment of the present invention;

[0039] Figure 6 It is a schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0040] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0041] Based on the problems of the existing tracking framework, this paper attempts to integrate the branches of the existing framework into a network to form a mutually beneficial positive feedback loop. Therefore, we designed an end-to-end multi-target tracking model that can identify the target position and information contained in each frame of the video. Figure 1-Figure 4 The multi-target tracking method of the present invention is described.

[0042] like Figure 1 As shown, this embodiment discloses a multi-target tracking method, including:

[0043] Step 101: Obtain a video to be tested;

[0044] Step 102: input the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0045] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0046] The multi-target tracking method of the embodiment of the present invention can be based on the enhanced features of historical trajectory features. The difference from traditional target detection is that the historical trajectory information is used to enhance the underlying features of the image to reduce false detection and missed detection, thereby greatly improving the detection accuracy of multi-target detection.

[0047] In at least one embodiment of the present invention, Figure 2 As shown, the end-to-end multi-target tracking model includes a feature enhancement module, a detection branch, a feature branch and an identity association branch;

[0048] The feature enhancement module is used to extract the underlying features of the current frame of the video to be tested, and enhance the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame;

[0049] The detection branch is used to obtain a detection frame where the target is located according to the enhanced features of the current frame;

[0050] The feature branch is used to obtain the detection frame feature of the current frame according to the enhanced feature of the current frame and the detection frame where the target is located;

[0051] The identity association branch is used to obtain the target identity information of the current frame by performing identity association according to the detection frame where the target of the current frame is located and the detection frame features and the historical trajectory features.

[0052] The multi-target tracking method of the embodiment of the present invention can transmit the trajectory information to the detection branch and the feature branch based on the trajectory-guided feature enhancement module to enhance the detection and feature branches, so that the identity association branch can generate high-quality trajectories. The different branches of multi-target tracking are integrated into a network to form a mutually beneficial positive feedback loop.

[0053] In at least one embodiment of the present invention, the step of enhancing the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame includes:

[0054] Obtaining the underlying features of the current frame and the historical trajectory features of the previous frame;

[0055] The continuous area between the underlying features of the current frame and the historical trajectory features of the previous frame is enhanced, and the interference area between the underlying features of the current frame and the historical trajectory features of the previous frame is erased to obtain the enhanced features of the current frame.

[0056] It should be noted that the feature enhancement module includes a trajectory-guided enhanced attention module and an erasing attention module.

[0057] Specifically, for the tth frame of any video, such as Figure 3 As shown, the feature enhancement module is used to give a candidate suggestion box and the original underlying features F t,* , as shown in Formula 1, using the historical trajectory feature map of the previous frame Generate Enhancement Features

[0058]

[0059] in, and Represents element addition and element subtraction operations, and Represents F t,* and Between the enhanced attention and erased attention, P t,* It is obtained by merging the public detection results with the target tracked in the previous frame, where the public detection results are the detection results provided by the dataset.

[0060] It should be noted that the feature enhancement module for trajectory guidance in the embodiment of the present invention utilizes the historical trajectory feature map Enhanced candidate suggestion box underlying characteristics.

[0061] In at least one embodiment of the present invention, the enhanced attention module f(·) is used to utilize the trajectory feature map Enhancement With F t,* The consistency region between . Given feature F t,* ∈R C×H×W and trajectory feature map collection The corresponding enhanced attention can be calculated by formula 2:

[0062]

[0063] in, Represents the feature map of the k-th trajectory, Represents F t,* With E t-1,k Random features in . θ, ρ represents the convolutional layer. λ*,k is the indicator function which can be calculated by Equation 3:

[0064]

[0065] Among them, IoU (P t,* ,B t-1,k ) represents the candidate suggestion box P t,* With trajectory The end coordinate B t-1,k The geometric similarity of iou is the geometric threshold.

[0066] In at least one embodiment of the present invention, the erasing attention module g(·) is used to eliminate the influence of background information and use the trajectory features to erase the underlying features F t,* The distractor area in the image, the corresponding attention elimination can be calculated by formula 4:

[0067]

[0068] Among them, e t-1,k The trajectory feature map E t-1,k The trajectory features obtained by the global average pooling layer, and Represents the relevant features and the binary mask. The relevant features represent the dot product similarity of all local features in et-1,k and Ft,*, which can be calculated by formula 5:

[0069]

[0070] Among them, F t,*[i,j] Represents F t,* Local features at spatial position [i, j] in the feature.

[0071] It should be noted that after obtaining relevant features, the feature enhancement module uses Generate F t,* With e t-1,k The binary mask between The binary mask Used to eliminate distracting areas.

[0072] In this embodiment, a mask is generated using a block binarization layer to erase F t,* The interference area. A sliding block of size 3×3 and stride 1 is used to search for the most prominent continuous area in the correlation feature, and the correlation value of each block is defined as the sum of the values ​​in the block. Then, the candidate block with the highest correlation value is selected as the block to be retained, that is, the binary mask of the correlation feature is obtained by setting the value of the selected block to 0 and the other values ​​to 1.

[0073] In at least one embodiment of the present invention, obtaining a detection frame where the target is located according to the enhanced features of the current frame includes:

[0074] According to the enhanced features of the t-th frame image Perform target detection and the corresponding detection results As shown in Formula 6, the detector generate:

[0075]

[0076] It should be noted that this embodiment is based on the Tracktor algorithm and uses the two-stage detector Faster R-CNN as the detection branch. Different from the traditional Faster R-CNN method of generating candidate suggestion boxes, the public detection result of the current frame is combined with the tracking result of the previous frame as the candidate suggestion box.

[0077] In at least one embodiment of the present invention, obtaining the detection frame feature of the current frame according to the enhanced feature of the current frame and the detection frame where the target is located includes:

[0078] Global pooling is performed based on the enhanced features of the current frame and the detection frame where the target of the current frame is located to obtain the detection frame features of the current frame, wherein the detection frame where the target of the current frame is located has a one-to-one correspondence with the detection frame features of the current frame.

[0079] It should be noted that when the detection branch predicts the current frame detection result Afterwards, the feature branch aims to generate corresponding robust features. Since trajectory features contain rich temporal complementary information, they can be used to enhance the description of occluded targets and reduce identity jumps. Therefore, trajectory features are used to enhance the feature description of detection results.

[0080] For the t-th frame image, given the enhanced features With detection box B t,i , the detection frame features of the current frame are shown in Formula 7:

[0081]

[0082] Among them, GAP represents the global pooling layer, ε represents the convolutional layer, and the current frame detection result The corresponding features are represented as

[0083] In at least one embodiment of the present invention, obtaining the target identity information of the current frame by performing identity association based on the detection frame where the target of the current frame is located and the detection frame features and the historical trajectory features includes:

[0084] Generate a detection map by using the detection frame features of the current frame corresponding to the detection frame coordinates of the target in the current frame as nodes and the spliced ​​features of the detection frame features of the current frame as edges;

[0085] The trajectory features corresponding to the end coordinates of the historical trajectory of the previous frame are used as nodes, and the concatenated features of the trajectory features of the previous frame are used as edges to generate a trajectory graph;

[0086] Graph matching is performed based on the detection graph and the trajectory graph to obtain the target identity information of the current frame.

[0087] It should be noted that, in the embodiment of the present invention, the identity association problem is converted into a detection result. And historical trajectory The graph matching problem between n and m represent the number of detection results and historical trajectories respectively.

[0088] The workflow of the identity association branch in the embodiment of the present invention includes the following sub-steps:

[0089] Step 201: construct a detection map and a trajectory map to describe the relationship between different targets between the detection results and the historical trajectory;

[0090] Step 202: Enhance node features by transferring cross-graph information between the detection graph and the trajectory graph.

[0091] Step 203: The graph matching layer uses high-order context information to match the detection graph and the trajectory graph.

[0092] It should be noted that the construction process in step 201 is as follows: for the t-th frame image, the detection image is defined as V D ={(B t,i ;f t,i )}(i∈[1,n]) represents a node set, where f t,i Represents the bounding box B t,i characteristics. represents the edge set, where is the corresponding node feature and Similarly, the trajectory graph is defined as V T ={(B t-1,j ;e t-1,j )}(j∈[1,m]), where B t-1,j Represents the jth historical trajectory The ending coordinates, e t-1,j is the trajectory feature. in represent and splicing features.

[0093] In at least one embodiment of the present invention, the step of performing graph matching based on the detection graph and the trajectory graph to obtain the target identity information of the current frame includes:

[0094] The detection image and the trajectory image are integrated with node features, a similarity matrix of the detection image and the trajectory image after the node feature integration is calculated, and target identity information of the current frame is obtained according to the similarity matrix.

[0095] It should be noted that the node feature integration method of this embodiment can simultaneously consider the influence of visual information and spatial information, and can make the network focus on nodes with higher similarity between the detection graph and the trajectory graph. Specifically, the node feature integration method of this embodiment is shown in Formula 8:

[0096]

[0097] in, and are the initial node features of the detection graph and trajectory graph respectively, and represents the features of the lth iteration, Represents a learnable function for any node v, e.g., MLP. Detection graph The final node feature is defined as Where L represents the number of information transmissions, that is, the number of times the node feature integration method of this embodiment updates the node features. The node integration method is also as shown in Formula 8.

[0098] It should be noted that obtaining the target identity information of the current frame according to the similarity matrix includes:

[0099] Computational detection graph With trajectory diagram The similarity matrix M between them;

[0100] Where M = ζ(M n ,M e ), M n is the first-order node similarity, M e is the second-order edge similarity, and ζ(·) represents the similarity matrix decomposition. n With M e Each element in is expressed as Equation 9:

[0101]

[0102] Among them, M n[i,j] represents the cosine similarity between the i-th node feature in the detection graph and the j-th node feature in the trajectory graph, M e {i 1 i 2 ,j 1 j 2} represents the cosine similarity between edge features.

[0103] It should be noted that in step 203, the image matching layer obtains the best matching result Y by using the matching formula in formula 10. * :

[0104] Y * = argmax Y Y T MY Type 10

[0105] The embodiment of the present invention conducts research experiments on four public data sets, namely MOT15, MOT16, MOT17 and MOT20, to prove the multi-target detection effect of the present invention.

[0106] The experimental results of the present invention and other methods on four data sets are shown in Table 1:

[0107] Table 1

[0108]

[0109]

[0110] It can be seen that the method of the present invention has a significant effect on the accuracy of multi-target tracking.

[0111] Figure 4 is a representative case obtained by comparing the method according to an embodiment of the present invention with other frameworks. Figure 4 It can be seen that the present invention improves the ability of multi-target tracking algorithms to solve occlusion and false detection. Compared with the other two frameworks, the end-to-end model constructed by the present invention based on trajectory features can alleviate false detection and missed detection caused by occlusion in complex scenes, such as Figure 4 The detection box marked by the arrow in the middle is the target that is not detected by the above two frameworks.

[0112] The multi-target tracking device provided by the present invention is described below. The multi-target tracking device described below and the multi-target tracking method described above can be referred to in correspondence with each other. Figure 5 As shown, the multi-target tracking device according to an embodiment of the present invention includes:

[0113] The acquisition module 501 is used to acquire the video to be tested;

[0114] An output module 502 is used to input the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0115] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0116] The multi-target tracking device of the embodiment of the present invention can enhance the features based on historical trajectory features. The difference from traditional target detection is that the historical trajectory information is used to enhance the underlying features of the image to reduce false detection and missed detection, thereby greatly improving the detection accuracy of multi-target detection.

[0117] In at least one embodiment of the present invention, the end-to-end multi-target tracking model includes a feature enhancement module, a detection branch, a feature branch, and an identity association branch;

[0118] The feature enhancement module is used to extract underlying features, and enhance the underlying features according to the historical trajectory features to obtain enhanced features;

[0119] The detection branch is used to obtain a detection frame where the target is located according to the enhanced features;

[0120] The feature branch is used to obtain the detection frame feature according to the enhanced feature and the detection frame where the target is located;

[0121] The identity association branch is used to obtain target identity information by performing identity association based on the detection frame where the target is located and the detection frame features and historical trajectory features.

[0122] In at least one embodiment of the present invention, the step of enhancing the underlying features according to the historical trajectory features to obtain enhanced features, for the current frame, includes:

[0123] Obtaining the underlying features of the current frame and the historical trajectory features of the previous frame;

[0124] The continuous area between the underlying features of the current frame and the historical trajectory features of the previous frame is enhanced, and the interference area between the underlying features of the current frame and the historical trajectory features of the previous frame is erased to obtain the enhanced features of the current frame.

[0125] In at least one embodiment of the present invention, obtaining the detection frame feature according to the enhanced feature and the detection frame where the target is located, for the current frame, includes:

[0126] Global pooling is performed based on the enhanced features of the current frame and the detection frame where the target of the current frame is located to obtain the detection frame features of the current frame, wherein the detection frame where the target of the current frame is located has a one-to-one correspondence with the detection frame features of the current frame.

[0127] In at least one embodiment of the present invention, the target identity information is obtained by performing identity association based on the detection frame where the target is located and the detection frame features and the historical trajectory features, and for the current frame, includes:

[0128] Generate a detection map by using the detection frame features of the current frame corresponding to the detection frame coordinates of the target in the current frame as nodes and the spliced ​​features of the detection frame features of the current frame as edges;

[0129] The trajectory features corresponding to the end coordinates of the historical trajectory of the previous frame are used as nodes, and the concatenated features of the trajectory features of the previous frame are used as edges to generate a trajectory graph;

[0130] Graph matching is performed based on the detection graph and the trajectory graph to obtain the target identity information of the current frame.

[0131] In at least one embodiment of the present invention, the step of performing graph matching based on the detection graph and the trajectory graph to obtain the target identity information of the current frame includes:

[0132] The detection image and the trajectory image are integrated with node features, a similarity matrix of the detection image and the trajectory image after the node feature integration is calculated, and target identity information of the current frame is obtained according to the similarity matrix.

[0133] Figure 6 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 6 As shown, the electronic device may include: a processor 610, a communication interface 620, a memory 630 and a communication bus 640, wherein the processor 610, the communication interface 620 and the memory 630 communicate with each other through the communication bus 640. The processor 610 may call the logic instructions in the memory 630 to execute the multi-target tracking method, which includes:

[0134] Get the video to be tested;

[0135] Inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0136] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0137] In addition, the logic instructions in the above-mentioned memory 630 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program codes.

[0138] On the other hand, the present invention further provides a computer program product, the computer program product includes a computer program, the computer program can be stored on a non-transitory computer-readable storage medium, when the computer program is executed by a processor, the computer can execute the multi-target tracking method provided by the above methods, the method includes:

[0139] Get the video to be tested;

[0140] Inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0141] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0142] In another aspect, the present invention further provides a non-transitory computer-readable storage medium having a computer program stored thereon, which is implemented when the computer program is executed by a processor to perform the multi-target tracking method provided by the above methods, the method comprising:

[0143] Get the video to be tested;

[0144] Inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model;

[0145] Among them, the multi-target information includes the detection box where the target is located and the target identity information, the end-to-end multi-target model is trained based on a video sample data set, and the features extracted during the training process of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set.

[0146] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Those of ordinary skill in the art may understand and implement it without creative effort.

[0147] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0148] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A multi-target tracking method, characterized in that: include: Get the video to be tested; Inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model; The multi-target information includes the detection frame where the target is located and the target identity information, the end-to-end multi-target model is obtained by training based on the video sample data set, and the features extracted during the training of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set; The end-to-end multi-target tracking model includes a feature enhancement module, a detection branch, a feature branch and an identity association branch; the feature enhancement module is used to extract the underlying features of the current frame of the video to be tested, and enhance the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame; the detection branch is used to obtain the detection frame of the target according to the enhanced features of the current frame; the feature branch is used to obtain the detection frame features of the current frame according to the enhanced features of the current frame and the detection frame of the target; the identity association branch is used to obtain the target identity information of the current frame according to the detection frame of the target in the current frame and the detection frame features and the historical trajectory features; The step of enhancing the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame includes: Obtaining the underlying features of the current frame and the historical trajectory features of the previous frame; The continuous area between the underlying features of the current frame and the historical trajectory features of the previous frame is enhanced, and the interference area between the underlying features of the current frame and the historical trajectory features of the previous frame is erased to obtain the enhanced features of the current frame.

2. The multi-target tracking method according to claim 1, characterized in that: The step of inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model comprises: Extracting underlying features of a current frame of the video to be tested, and enhancing the underlying features according to historical trajectory features to obtain enhanced features of the current frame; Obtaining a detection frame where the target is located according to the enhanced features of the current frame; Obtaining a detection frame feature of the current frame according to the enhanced feature of the current frame and the detection frame where the target is located; The target identity information of the current frame is obtained by performing identity association based on the detection frame where the target of the current frame is located and the detection frame features and the historical trajectory features.

3. The multi-target tracking method according to claim 1, characterized in that: The obtaining the detection frame feature of the current frame according to the enhanced feature of the current frame and the detection frame where the target is located includes: Global pooling is performed based on the enhanced features of the current frame and the detection frame where the target of the current frame is located to obtain the detection frame features of the current frame, wherein the detection frame where the target of the current frame is located has a one-to-one correspondence with the detection frame features of the current frame.

4. The multi-target tracking method according to claim 3, characterized in that: The obtaining the target identity information of the current frame by performing identity association according to the detection frame where the target of the current frame is located and the detection frame features and the historical trajectory features includes: Generate a detection map by using the detection frame features of the current frame corresponding to the detection frame coordinates of the target in the current frame as nodes and the spliced ​​features of the detection frame features of the current frame as edges; The trajectory features corresponding to the end coordinates of the historical trajectory of the previous frame are used as nodes, and the concatenated features of the trajectory features of the previous frame are used as edges to generate a trajectory graph; Graph matching is performed based on the detection graph and the trajectory graph to obtain the target identity information of the current frame.

5. The multi-target tracking method according to claim 4, characterized in that: The obtaining target identity information of the current frame by performing graph matching according to the detection graph and the trajectory graph includes: The detection image and the trajectory image are integrated with node features, a similarity matrix of the detection image and the trajectory image after the node feature integration is calculated, and target identity information of the current frame is obtained according to the similarity matrix.

6. A multi-target tracking device, characterized in that: include: An acquisition module is used to obtain the video to be tested; An output module, used for inputting the video to be tested into an end-to-end multi-target tracking model to obtain multi-target information contained in the video to be tested output by the end-to-end multi-target tracking model; The multi-target information includes the detection frame where the target is located and the target identity information, the end-to-end multi-target model is obtained by training based on the video sample data set, and the features extracted during the training of the end-to-end multi-target model are enhanced features based on historical trajectory features extracted from the video sample data set; The end-to-end multi-target tracking model includes a feature enhancement module, a detection branch, a feature branch and an identity association branch; the feature enhancement module is used to extract the underlying features of the current frame of the video to be tested, and enhance the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame; the detection branch is used to obtain the detection frame of the target according to the enhanced features of the current frame; the feature branch is used to obtain the detection frame features of the current frame according to the enhanced features of the current frame and the detection frame of the target; the identity association branch is used to obtain the target identity information of the current frame according to the detection frame of the target in the current frame and the detection frame features and the historical trajectory features; The step of enhancing the underlying features according to the historical trajectory features to obtain the enhanced features of the current frame includes: Obtaining the underlying features of the current frame and the historical trajectory features of the previous frame; The continuous area between the underlying features of the current frame and the historical trajectory features of the previous frame is enhanced, and the interference area between the underlying features of the current frame and the historical trajectory features of the previous frame is erased to obtain the enhanced features of the current frame.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the multi-target tracking method according to any one of claims 1 to 5 are implemented.

8. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multi-target tracking method according to any one of claims 1 to 5 are implemented.

9. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the multi-target tracking method according to any one of claims 1 to 5 are implemented.

Images