Joint detection tracking method and device, electronic equipment and storage medium

Abstract

The invention discloses a joint detection tracking method and device, electronic equipment and a storage medium, and relates to the technical field of target tracking, and the method comprises the steps: carrying out the training through intra-frame direct matching and inter-frame indirect matching, calculating the self-comparison loss, and obtaining a discrimination representation; adjusting the matching similarity by using the consistency of cross-frame matching and continuous matching, calculating cross comparison loss, and extracting features beneficial to target association; meanwhile, shielding, disappearing and reappearing targets are considered, fuzzy targets are matched again, fuzzy comparison loss is calculated, and the certainty of follow-up target association is improved; synthesizing the self-comparison loss, the cross-comparison loss and the fuzzy comparison loss, and calculating the total loss; and tracking and identifying the target according to the total loss. According to the joint detection tracking method and device, the electronic equipment and the storage medium provided by the invention, the target in the camera range can be detected and tracked under the condition that a large amount of labeled data is not used.

Description

Technical Field

[0001] The present application relates to the field of target tracking technology, and in particular to a joint detection and tracking method, a joint detection and tracking device, an electronic device and a storage medium. Background Art

[0002] In daily life, traffic scenes are relatively complex. The movement of various targets around the vehicle during driving may affect the vehicle's driving, such as a person running suddenly. Target tracking can not only provide the target's motion trajectory information, but also the subsequent system can judge whether an abnormal situation occurs based on this information. Once an abnormality occurs, an alarm message will be issued immediately to remind people to pay attention to the corresponding area to avoid unnecessary losses. At present, advanced tracking methods are trained based on data labels, but the scenes and targets are diverse, and it is difficult and expensive to obtain a large amount of labeled supervision data. Clustering methods are prone to accumulate errors during training. And as a video task, the targets in multi-target tracking are constantly changing over time, which leads to the inevitable mutual occlusion between targets and targets, between targets and non-targets, and the disappearance of old targets and the appearance of new targets. Occlusion will directly reduce the characteristics of the target itself and introduce additional features from other targets or non-targets, making the representation of the occluded target inconsistent between different frames, reducing the similarity of the same target, and having a negative impact on data association. For disappeared and newly appeared targets, these targets cannot be matched successfully with other targets in theory, so it can be said that the entire association stage is almost negative. These problems are the biggest challenges that researchers need to face even in supervised multi-target tracking. Most of the work is to post-process the related targets in the data association stage, and rarely directly balance the features of all targets at the same time in the training feature stage. Moreover, in the unsupervised case, the dual limitations of no labels and the inherent problems of multi-target tracking make unsupervised multi-target tracking more difficult. Summary of the invention

[0003] The object of the present invention is to provide a joint detection and tracking method, device, electronic device and storage medium for detecting and tracking targets within the range of a camera without using a large amount of labeled data.

[0004] The present invention provides the following scheme:

[0005] According to one aspect of the present invention, a joint detection and tracking method is provided, the joint detection and tracking method comprising:

[0006] Through training through direct intra-frame matching and indirect inter-frame matching, the self-comparison loss is calculated to obtain discriminative representations;

[0007] Using the consistency of cross-frame matching and continuous matching, the matching similarity is adjusted, the cross-matching loss is calculated, and features that are beneficial to target association are extracted;

[0008] At the same time, the occluded, disappeared, and reappeared targets are considered, and the fuzzy comparison loss is calculated by re-matching these fuzzy targets to improve the certainty of subsequent target association;

[0009] Combine self-contrast loss, cross-contrast loss and fuzzy contrast loss to calculate the total loss;

[0010] According to the total loss, the target is tracked and identified.

[0011] Optionally, the training is performed by direct intra-frame matching and indirect inter-frame matching, and the self-comparison loss is calculated to obtain a discriminative representation, including:

[0012] Use the current frame feature matrix to directly calculate the similarity of objects within the frame;

[0013] By performing self-comparative learning on multiple frames, the similarity between objects can be further measured;

[0014] The self-alignment loss is calculated based on the similarity of objects within a frame and the similarity between objects.

[0015] Optionally, the calculating the self-comparison loss according to the similarity of the objects in the frame and the similarity between the objects includes:

[0016] The self-contrast loss is calculated according to the following formula:

[0017]

[0018] Among them, diag() means obtaining the diagonal matrix operation, S dsc is the matching similarity matrix of direct self-alignment, S isc is the matching similarity matrix of indirect self-alignment.

[0019] Optionally, the using the current frame feature matrix to directly calculate the intra-frame object similarity includes:

[0020] By calculating the similarity between the target feature and other target features, a similarity matrix is ​​formed;

[0021] The first matching similarity matrix is ​​calculated by using the softmax operation on the similarity matrix.

[0022] Optionally, the similarity between the objects is further measured by performing self-comparison learning on multiple frames, including:

[0023] According to the circular correlation consistency, the forward correlation matrix and the reverse correlation matrix are calculated;

[0024] A second matching similarity matrix is ​​calculated according to the forward correlation matrix and the reverse correlation matrix.

[0025] Optionally, utilizing the consistency of cross-frame matching and continuous matching, adjusting the matching similarity, calculating the cross-matching loss, and extracting features that are beneficial to target association include:

[0026] Calculate the matching similarity matrix between different frames;

[0027] According to the matching similarity matrix between different frames, the JS divergence between different frames is calculated;

[0028] According to the JS divergence, the cross comparison loss between different frames is calculated.

[0029] Optionally, calculating the cross comparison loss between different frames according to the JS divergence includes:

[0030] The cross-matching loss is calculated according to the following formula:

[0031]

[0032] Among them, S 1→3 represents the similarity matrix of the target in frame 1 matching frame 3, S 3→1 Represents the similarity matrix of the target in frame 3 matching frame 1, Represents the matching similarity matrix of the target in frame 1 matched to frame 3, Represents the matching similarity matrix of the target in frame 3 matched to frame 1.

[0033] Optionally, the method of simultaneously considering occluded, disappeared, and reappeared targets, re-matching these fuzzy targets, and calculating the fuzzy comparison loss to improve the certainty of subsequent target association includes:

[0034] Find the blurred target in the first frame based on similarity;

[0035] Find the blurred target in the second frame based on the similarity;

[0036] The similarity of these fuzzy targets is calculated again to obtain a similarity matrix.

[0037] Optionally, the similarity of these fuzzy targets is calculated again to obtain a similarity matrix, including:

[0038] The similarity matrix is ​​calculated according to the following formula:

[0039]

[0040] in, and The similarity of these fuzzy targets is calculated again according to the formula to obtain the similarity matrix, N r and Mr are the number of blurred objects in frame 1 and frame 2, respectively.

[0041] Optionally, the comprehensive self-contrast loss, cross-contrast loss and fuzzy contrast loss are used to calculate the total loss, including:

[0042] The total loss is calculated according to the following formula:

[0043] L un_reid =L(I t ,I t-1 ,I t-2 )=L sc +L cc +L ac

[0044] Among them, I t ,I t-1 ,I t-2 Represents three consecutive frames as input, L un_reid represents the total loss, L sc represents the self-alignment loss, L cc represents the cross-matching loss, L ac Represents the fuzzy comparison loss.

[0045] According to two aspects of the present invention, a joint detection and tracking device is provided, the joint detection and tracking device comprising:

[0046] The self-alignment module is used to train through direct intra-frame matching and indirect inter-frame matching, calculate the self-alignment loss, and obtain discriminative representations;

[0047] The cross-matching module is used to utilize the consistency of cross-frame matching and continuous matching, calculate the cross-matching loss, adjust the matching similarity, and extract features that are beneficial to target association;

[0048] The fuzzy comparison module is used to simultaneously consider occluded, disappeared, and reappearing targets, and to improve the certainty of subsequent target association by re-matching these fuzzy targets and calculating the fuzzy comparison loss;

[0049] The loss calculation module is used to combine the self-contrast loss, cross-contrast loss and fuzzy contrast loss to calculate the total loss;

[0050] The tracking and identification module is used to track and identify the target based on the total loss.

[0051] According to three aspects of the present invention, there is provided an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0052] A computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the joint detection and tracking method.

[0053] According to four aspects of the present invention, a computer-readable storage medium is provided, comprising: a computer program executable by an electronic device is stored therein, and when the computer program runs on the electronic device, the electronic device executes the steps of the joint detection and tracking method.

[0054] Through the above scheme, the following beneficial technical effects are obtained:

[0055] The present invention provides a joint detection and tracking method based on unsupervised learning, which detects and tracks targets within the camera range without using a large amount of labeled data. Inspired by contrastive learning and cycle consistency theory in unsupervised learning, contrastive learning is introduced and a corresponding unsupervised method is proposed for the case where the data set lacks ID labels. Self-supervisory signals (self-contrast, cross-contrast, fuzzy contrast) are constructed to promote the model to extract discriminative features;

[0056] The self-comparison module is trained through direct matching within frames and indirect matching between frames to obtain more discriminative representations;

[0057] The cross comparison module uses the consistency of cross-frame matching and continuous matching to adjust the matching similarity and extract features that are more beneficial to target association;

[0058] The fuzzy contrast module simultaneously considers occluded, disappeared, and reappearing targets, and further improves the certainty of subsequent target association by re-matching these fuzzy targets;

[0059] Finally, the above three self-supervised modules are combined to complete self-supervised multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0060] Figure 1 This is a comparison diagram of the principles between supervised learning and unsupervised learning.

[0061] Figure 2 It is a flowchart of a joint detection and tracking method provided by one or more embodiments of the present invention.

[0062] Figure 3 is a flow chart of a self-alignment process provided by one or more embodiments of the present invention.

[0063] Figure 4 It is a flowchart of intra-frame target similarity calculation provided by one or more embodiments of the present invention.

[0064] Figure 5 is a flowchart of inter-frame target similarity calculation provided by one or more embodiments of the present invention.

[0065] Figure 6 It is a flow chart of the cross comparison process provided by one or more embodiments of the present invention.

[0066] Figure 7 is a flow chart of a fuzzy comparison process provided by one or more embodiments of the present invention.

[0067] Figure 8 It is an overall framework diagram of the joint detection and tracking method provided by one or more embodiments of the present invention.

[0068] Fig. 9 It is a schematic diagram of the principle of self-alignment and cross-alignment provided by one or more embodiments of the present invention.

[0069] Fig.10 It is a schematic diagram of the fuzzy comparison provided by one or more embodiments of the present invention.

[0070] Fig.11 It is a structural diagram of a joint detection and tracking device provided by one or more embodiments of the present invention.

[0071] Fig.12 is a structural diagram of an electronic device provided by one or more embodiments of the present invention. DETAILED DESCRIPTION

[0072] The technical solution of the present invention will be described clearly and completely below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0073] Unsupervised multi-target tracking mainly involves two major technologies: unsupervised learning and multi-target tracking. The main goal of the multi-target tracking algorithm is to continuously track all targets of interest in continuous video frames and obtain their temporal trajectories without knowing the number of targets in advance. Multi-target tracking is a further extension of target detection. Unlike target detection, which focuses on obtaining the location information (bounding box) of each target, multi-target tracking requires assigning a unique ID to each target based on target detection in order to distinguish different targets in the video sequence and maintain the trajectory of each target. The most commonly used tracking method is detection-based:

[0074] ① Target detection: Use advanced target detection algorithms to process each frame of the image to obtain relevant information such as the location of all targets in the frame.

[0075] ② Feature extraction: Input the targets obtained by target detection into the feature extraction network to obtain the discriminative features of each target. These features are crucial for subsequent target association.

[0076] ③Similarity calculation: Calculate the similarity between objects in different frames based on their appearance and position features. This helps determine the degree of association between objects in different frames.

[0077] ④ Target association: Using the results of similarity calculation, the data association algorithm is used to match the target. For the successfully matched target, the corresponding ID is assigned to it and its track is maintained; for the unmatched detection results, it is considered to be a new target and a new track is initialized for it; for the unmatched track, the target is considered to have disappeared.

[0078] However, the multi-target tracking algorithm that separates detection and re-identification means that at least two computationally intensive components are required in the system: the detector and the re-identification model. This leads to some problems: computational complexity, difficulty of end-to-end training, and the impact of the number of targets.

[0079] Unsupervised learning. Supervised learning requires a large amount of labeled data for training in many tasks. Figure 1 As shown in the left branch, in the supervised case, it can be treated as a classification task. However, obtaining a large amount of labeled supervised data is a difficult and expensive task. In contrast, unsupervised learning can not only overcome the lack of labeled data, but also better adapt to new fields and scenarios. Some methods have demonstrated that even in the absence of labels, the network training can be carried out in the true direction.

[0080] Figure 1 is a comparison of supervised and unsupervised methods. Figure 1 In unsupervised multi-target tracking, clustering is basically used, such as bottom-up clustering, which directly optimizes the cosine distance between samples or clusters through repelled loss to maximize the diversity of targets within the same category while maximizing the similarity of targets with the same ID. Figure 1 As shown in the right branch, objects with high similarity are considered to be of the same class, and pseudo labels are generated for network training. However, clustering methods are prone to accumulate errors during the training process.

[0081] The present invention provides a joint detection and tracking method based on unsupervised learning, which detects and tracks targets within the camera range without using a large amount of labeled data. Inspired by contrastive learning and cycle consistency theory in unsupervised learning, contrastive learning is introduced and a corresponding unsupervised method is proposed for the case where the data set lacks ID labels. Self-supervisory signals (self-contrast, cross-contrast, fuzzy contrast) are constructed to promote the model to extract discriminative features.

[0082] ① The self-comparison module is trained through direct matching within frames and indirect matching between frames to obtain a more discriminative representation.

[0083] ② The cross-comparison module uses the consistency of cross-frame matching and continuous matching to adjust the matching similarity and extract features that are more beneficial to target association.

[0084] ③ The fuzzy contrast module simultaneously considers occluded, disappeared, and reappearing targets, and further improves the certainty of subsequent target associations by re-matching these fuzzy targets.

[0085] Finally, the above three self-supervised modules are combined to complete self-supervised multi-target tracking.

[0086] Figure 2 is a flowchart of a joint detection and tracking method provided by one or more embodiments of the present invention. Figure 2 , the joint detection and tracking method includes the following steps:

[0087] S21, trains through direct intra-frame matching and indirect inter-frame matching, calculates the self-comparison loss, and obtains discriminative representations.

[0088] S22, using the consistency of cross-frame matching and continuous matching, adjusts the matching similarity, calculates the cross-matching loss, and extracts features that are beneficial to target association.

[0089] S23, taking into account the occluded, disappeared, and reappeared targets, by re-matching these blurred targets and calculating the fuzzy comparison loss, the certainty of subsequent target association is improved.

[0090] S24, integrates self-contrast loss, cross-contrast loss and fuzzy contrast loss to calculate the total loss.

[0091] S25, tracking and identifying the target according to the total loss.

[0092] In the unsupervised target tracking solution provided by the present application, there are three types of feature extraction methods: the first type is feature extraction within a frame, the second type is cross-frame, or inter-frame feature extraction, and the last type is fuzzy feature extraction.

[0093] The three different feature extraction methods have their own characteristics. For the first type of feature extraction, the feature matching of the target itself in the same frame is trained to obtain a discriminative representation.

[0094] The first type of feature extraction includes two types of operations: intra-frame direct matching and inter-frame indirect matching. The so-called intra-frame direct matching refers to the direct comparison learning between targets in the same frame by calculating the similarity between the target features and other target features, which emphasizes the similarity between targets and provides useful information for subsequent association.

[0095] The indirect matching between frames refers to further measuring the similarity between targets by performing self-comparison learning on multiple frames.

[0096] The first type of feature extraction mentioned above effectively reduces interference from other targets by maximizing its own similarity.

[0097] The feature extraction of the second category also includes two specific categories. They are: cross-frame matching and continuous matching. Regardless of which of the above two categories, the similarity characteristics between targets in different frames are used to adjust the matching similarity to extract features that are more beneficial to target association. This branch considers the matching relationship between targets in different frames, allowing the model to better understand the target's motion pattern and appearance changes, thereby improving the accuracy of target association.

[0098] For the third type of feature extraction, occlusion, disappearance, and reappearance of targets are considered at the same time. By re-matching these blurred targets, the certainty of subsequent target association is further improved. In the case of occlusion, disappearance, reappearance, etc. of the target, the third type of feature extraction can significantly improve the robustness of target association.

[0099] Regardless of which of the three categories of feature extraction is used, the result is the calculation result of the loss function under that branch. Specifically, in the self-matching operation branch, the final result obtained is the calculation result of the loss function of the self-matching. In the cross-matching operation branch, the final result obtained is the calculation result of the loss function of the cross-matching. In the fuzzy matching operation branch, the final result obtained is the calculation result of the loss function of the fuzzy matching.

[0100] After waiting for the loss functions of the three branches to obtain the calculated results, the self-matching loss, cross-matching loss, and fuzzy matching loss are combined to obtain the total loss of the target. Finally, the target is tracked and identified based on the total loss of the three branches.

[0101] Since the target tracking and recognition process mentioned above uses the similarity of targets within a frame, the similarity of targets between frames, and the calculation method of fuzzy comparison, the constructed self-supervisory signal includes a self-comparison signal, a cross-comparison signal, and a fuzzy comparison signal, and the discriminant row features of the model are enhanced, so that the target in the image can still be accurately tracked without supervised data.

[0102] In particular, since the fuzzy comparison calculation method is specially adopted in the three branches, the model can cope with a series of complex situations such as target occlusion, disappearance, reappearance, etc., and the target tracking of the model has stronger robustness.

[0103] The FairMOT framework (including a backbone network, a detection branch, and a re-identification feature extraction branch) is used as the baseline network. In the training phase, the three contrastive learning submodules are trained. The total loss function of the unsupervised learning part is as follows:

[0104] L un_reid =L(I t ,I t-1 ,I t-2 )=L sc +L cc +L ac

[0105] Among them, three consecutive frames I t ,I t-1 ,I t-2 As input, L sc , L cc and L ac They correspond to the self-contrast loss, cross-contrast loss and fuzzy contrast loss constructed above respectively.

[0106] In the inference test phase, video frames are input into the network frame by frame in order, and then the corresponding detection results and re-identification embeddings are obtained. First, the trajectory corresponding to each target is initialized using the detection results of the first frame, and then the data association is completed using two-stage matching. The overall association strategy is basically the same as FairMOT, using Kalman filtering to predict the location of the target and using appearance features to match the detection bounding box with the existing trajectory. For the trajectories and detections that are not matched in the first stage, the IOU distance is used again for secondary matching. Finally, the unmatched detections are initialized as new targets. At the same time, the unmatched trajectories are saved for 30 frames so that they can be matched when they appear again in subsequent frames.

[0107] Figure 3 is a flow chart of a self-comparison process provided by one or more embodiments of the present invention. Figure 3, training through intra-frame direct matching and inter-frame indirect matching, calculating the self-matching loss, and obtaining a discriminative representation includes the following steps:

[0108] S31, using the current frame feature matrix, directly calculate the intra-frame target similarity.

[0109] S32, further measures the similarity between objects by performing self-comparative learning on multiple frames.

[0110] S33, calculating the self-comparison loss according to the similarity of the objects within the frame and the similarity between the objects.

[0111] For the self-alignment operation, it is divided into two main modules: direct self-alignment module and indirect self-alignment module.

[0112] In the direct self-alignment module, the objects within the frame are used for mutual alignment. In the indirect self-alignment module, the objects between frames are used for mutual alignment.

[0113] In the direct self-alignment module, for each target, the similarity matrix S is formed by calculating the similarity between the target feature and other target features. ds By performing softmax normalization on each row of the similarity matrix, we get the matching similarity matrix S dsc , represents the matching probability of each target with other targets.

[0114] In the indirect self-comparison module, the similarity between objects is further measured by performing self-comparison learning on multiple frames.

[0115] After completing the operations of the direct self-alignment module and the indirect self-alignment module, the self-alignment loss is calculated using the operation results of the two main modules, that is, the similarity of the objects within the frame and the similarity between the objects. The calculation formula of the self-alignment loss is as follows:

[0116]

[0117] Among them, diag() is an operation to obtain the diagonal matrix.

[0118] Figure 4 This is a process for calculating the intra-frame object similarity provided by one or more embodiments of the present invention. Figure 4 , using the current frame feature matrix, directly calculating the intra-frame target similarity includes the following steps:

[0119] S41, forming a similarity matrix by calculating the similarity between the target feature and other target features.

[0120] S42, calculating a first matching similarity matrix by performing a softmax operation on the similarity matrix.

[0121] For each target, the similarity matrix S is formed by calculating the similarity between the target feature and other target features. ds The specific calculation formula is as follows:

[0122]

[0123] Among them, X1 is the current frame feature matrix, and

[0124] Then, through the similarity matrix S ds Perform a softmax operation and normalize. The result of the above softmax operation is the first matching similarity matrix S dsc The first matching similarity matrix S dsc Represents the matching probability of each target with other targets. This part allows the network to perform direct comparative learning between targets in the same frame, emphasizes the similarity between targets, and provides useful information for subsequent association.

[0125] The calculation formula of the first matching similarity matrix is ​​as follows:

[0126] S dsc =Ψ row (S ds )

[0127] Among them, row It is the softmax in row direction.

[0128] Figure 5 is a flowchart of inter-frame target similarity calculation provided by one or more embodiments of the present invention. Figure 5 , by performing self-comparison learning on multiple frames, further measuring the similarity between targets includes the following steps:

[0129] S51, calculating the forward correlation matrix and the reverse correlation matrix according to the circular correlation consistency.

[0130] S52, calculating a second matching similarity matrix according to the forward correlation matrix and the reverse correlation matrix.

[0131] Multi-target tracking itself is performed on multiple frames, so the similarity between targets is further measured by performing comparative learning on multiple frames.

[0132] Taking frame 1 and frame 2 as an example, the similarity matrix between objects in different frames is Similar to the above embodiment, the similarity matrix is ​​further calculated. The calculation formula of the similarity matrix is ​​as follows:

[0133] S 1→2 =Ψ row (S is )

[0134]

[0135] Among them, S 1→2 and S 2→1 The results of the target matching of frame 1 to frame 2 and the target matching of frame 2 to frame 1 are respectively corresponded. Specifically, S 1→2 and S 2→1 The elements in the i-th row and j-th column of the matrix are as follows:

[0136]

[0137] Among them, τ is the temperature hyperparameter.

[0138] According to the cyclic association consistency, after the forward association and the reverse association, each target should ideally be able to match back to itself, so:

[0139] S isc =S 1→2 S 2→1

[0140] The corresponding self-contrastive module loss can be expressed as:

[0141]

[0142] Among them, diag() is an operation to obtain the diagonal matrix.

[0143] Due to the self-comparison, it is obvious that the similarity between the same targets should be the largest. Therefore, it is expected that the diagonal elements in the matching similarity matrix and should be the largest and as close to 1 as possible.

[0144] The self-contrastive design enables the network to better learn the feature representation of the target, emphasizes the similarity between targets, and provides more discriminative features for multi-target tracking tasks, thereby improving the accuracy and robustness of the tracking algorithm.

[0145] Figure 6 is a flow chart of the cross comparison process provided by one or more embodiments of the present invention. Figure 6 , using the consistency of cross-frame matching and continuous matching, adjusting the matching similarity, calculating the cross-matching loss, and extracting features that are beneficial to target association include the following steps:

[0146] S61, calculating the matching similarity matrix between different frames.

[0147] S62, calculating the JS divergence between different frames according to the matching similarity matrix between different frames.

[0148] S63, calculate the cross comparison loss between different frames according to the JS divergence.

[0149] Specific as Figure 6 As shown, the three frames As input, the matching similarity matrix between different frames is calculated, namely S 1→2 , S 2→1 , S 1→3 , S 3→1 , S 2→3 , S 3→2 As shown in the second column of the figure, we use and to calculate the similarity matrix of the target matching of frame 3 to frame 1. Similarly, we use S 1→2 and S 2→3 To calculate the similarity matrix of the frame 1 target matching to frame 3:

[0150]

[0151] These matching matrices generated by the intermediate frames should be identical to the matching matrices generated directly.

[0152] To measure the difference between different matching distributions, relative entropy (KL divergence) is used. Although KL divergence is often used to calculate the difference between two distributions P and Q,

[0153]

[0154] However, due to its asymmetry, we further adopt the JS divergence with symmetric properties.

[0155]

[0156] Where T = (P + Q) / 2, the corresponding cross-contrast loss is as follows,

[0157]

[0158] By making the results of continuous and cross-frame matching close to each other, different matching results are used to alleviate the differences between the same targets caused by occlusion.

[0159] This module provides a more flexible mechanism when dealing with target occlusion. By considering the consistency between continuous and cross-frame matching, it helps to alleviate the matching uncertainty caused by occlusion and improves the robustness of the multi-target tracking algorithm.

[0160] Figure 7 is a flow chart of the fuzzy comparison process provided by one or more embodiments of the present invention. Figure 7, while considering the occluded, disappeared, and reappeared targets, by re-matching these fuzzy targets and calculating the fuzzy comparison loss, the certainty of subsequent target association is improved, including the following steps:

[0161] S71, searching for a blurred target in the first frame according to the similarity.

[0162] S72, searching for a blurred target in the second frame according to the similarity.

[0163] S73, performing similarity calculation on these fuzzy targets again to obtain a similarity matrix.

[0164] The unsupervised training process proposed for fuzzy targets is as follows Figure 7 As shown. According to the obtained S 1→2 The matching matrix of S is used to find the blurred target in frame 1 according to the similarity. 2→1 The matching results are used to obtain the blurred targets in frame 2, and then the similarity of these blurred targets is calculated again according to the formula to obtain the similarity matrix and Where and are the number of blurred objects in frame 1 and frame 2 respectively. Finally, the loss of this module is calculated by minimum entropy,

[0165]

[0166] Among them, the adaptive coefficient is used to dynamically adjust the impact of the two reasons mentioned above. When the number of blurred targets in the two frames is the same, since the two frames are very close, this article can directly assume that there is no disappearance or appearance of the target, only occlusion exists, and the entropy must be as small as possible. When the number of blurred targets in the two frames is not equal, it must include disappeared and newly appeared targets. These targets have a low similarity with other targets because they cannot be matched. At this time, the coefficient is used to weaken this loss.

[0167] Figure 8 FIG. 1 is a general framework diagram of the joint detection and tracking method provided by one or more embodiments of the present invention. Figure 8 The training core consists of three core modules, namely the self-matching module, the cross-matching module, and the fuzzy matching module. The three modules work together to track the target in the image. Moreover, due to the full use of various features in the three modules, it can easily cope with various situations where the target is blocked, disappears, or reappears, and is more robust.

[0168] In addition, during the inference test phase, video frames are input into the network frame by frame in order, and then the corresponding detection results and re-identification embeddings are obtained. First, the trajectory corresponding to each target is initialized using the detection result of the first frame, and then the data association is completed using two-stage matching.

[0169] Fig. 9 is a schematic diagram of the principle of self-alignment and cross-alignment provided by one or more embodiments of the present invention. Fig. 9 ,The self-comparison module mainly uses intra-frame features for comparison, completes the extraction of intra-frame features and the calculation of the loss function. ,The cross-comparison module mainly uses inter-frame features for comparison, completes the extraction of inter-frame features and the calculation of the loss function.

[0170] Fig.10 is a schematic diagram of the fuzzy comparison provided by one or more embodiments of the present invention. Fig.10 ,By re-matching these fuzzy targets, the certainty of subsequent target associations can be further improved.

[0171] like Fig.11 As shown, the present invention also provides a joint detection and tracking device, comprising:

[0172] The self-comparison module 111 is used to perform training through direct matching within a frame and indirect matching between frames, calculate the self-comparison loss, and obtain a discriminative representation.

[0173] The cross-matching module 112 is used to calculate the cross-matching loss, adjust the matching similarity, and extract features that are beneficial to target association by utilizing the consistency of cross-frame matching and continuous matching.

[0174] The fuzzy comparison module 113 is used to simultaneously consider occluded, disappeared, and reappearing targets, and to improve the certainty of subsequent target association by re-matching these fuzzy targets and calculating the fuzzy comparison loss.

[0175] The loss calculation module 114 is used to integrate the self-contrast loss, the cross-contrast loss and the fuzzy contrast loss to calculate the total loss.

[0176] The tracking and identifying module 115 is used to track and identify the target according to the total loss.

[0177] It is worth noting that although only some basic functional modules are disclosed in the embodiments of the present invention, it does not mean that the composition of the present system is limited to the above basic functional modules. On the contrary, what the present embodiment wants to express is that on the basis of the above basic functional modules, those skilled in the art can arbitrarily add one or more functional modules in combination with the prior art to form an infinite number of embodiments or technical solutions. In other words, the present system is open rather than closed. Just because the present embodiment only discloses individual basic functional modules, it cannot be considered that the protection scope of the claims of the present invention is limited to the disclosed basic functional modules. At the same time, for the convenience of description, the above devices are described in terms of functions and are described separately in various units and modules. Of course, when implementing the present invention, the functions of each unit and module can be implemented in the same or one or more software and / or hardware.

[0178] like Fig.12 As shown, the present invention also provides an electronic device, comprising: a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of the joint detection and tracking method.

[0179] Fig.12 Schematic diagram of the structure of an electronic device provided by an embodiment of the present invention. Fig.12 The electronic device provided in the embodiment of the present invention includes: one or more processors 1210 and a memory 1220; the processor 1210 in the electronic device may be one or more, Fig.12 A processor 1210 is taken as an example; the memory 1220 is used to store one or more programs; the one or more programs are executed by the one or more processors 1210, so that the one or more processors 1210 implement the joint detection and tracking method as described in any one of the embodiments of the present invention.

[0180] The electronic device may further include: an input device 1230 and an output device 1240 .

[0181] The processor 1210, memory 1220, input device 1230 and output device 1240 in the electronic device can be connected via a bus or other means. Fig.12 The example of connecting through bus is taken in the following.

[0182] The memory 1220 in the electronic device is a computer-readable storage medium that can be used to store one or more programs, which can be software programs, computer executable programs, and modules, such as the program instructions / modules corresponding to the full-link test method for voice vehicle control provided in the embodiment of the present invention. The processor 1210 executes various functional applications and data processing of the electronic device by running the software programs, instructions, and modules stored in the memory 1220, that is, implementing the joint detection and tracking method in the above method embodiment.

[0183] The memory 1220 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 1220 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 1220 may further include a memory remotely arranged relative to the processor 1210, and these remote memories may be connected to the device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0184] The input device 1230 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device. The output device 1240 may include a display device such as a display screen.

[0185] The present invention also provides a computer-readable storage medium storing a computer program executable by an electronic device. When the computer program runs on the electronic device, the electronic device executes the steps of the joint detection and tracking method.

[0186] Specifically, the computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0187] The present invention also provides a vehicle, wherein the vehicle is provided with the combined detection and tracking device as described above.

[0188] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A joint detection and tracking method, characterized in that: The joint detection and tracking method comprises: Through training through direct intra-frame matching and indirect inter-frame matching, the self-comparison loss is calculated to obtain discriminative representations; Using the consistency of cross-frame matching and continuous matching, the matching similarity is adjusted, the cross-matching loss is calculated, and features that are beneficial to target association are extracted; At the same time, the occluded, disappeared, and reappeared targets are considered, and the fuzzy comparison loss is calculated by re-matching these fuzzy targets to improve the certainty of subsequent target association; Combine self-contrast loss, cross-contrast loss and fuzzy contrast loss to calculate the total loss; According to the total loss, the target is tracked and identified.

2. The method according to claim 1, characterized in that: The training is performed by direct intra-frame matching and indirect inter-frame matching, and the self-comparison loss is calculated to obtain a discriminative representation, including: Use the current frame feature matrix to directly calculate the similarity of objects within the frame; By performing self-comparative learning on multiple frames, the similarity between objects can be further measured; The self-alignment loss is calculated based on the similarity of objects within a frame and the similarity between objects.

3. The method according to claim 2, characterized in that The self-comparison loss is calculated based on the similarity of the objects in the frame and the similarity between the objects, including: The self-contrast loss is calculated according to the following formula: Among them, diag() means obtaining the diagonal matrix operation, S dsc is the matching similarity matrix of direct self-alignment, S isc is the matching similarity matrix of indirect self-alignment.

4. The method according to claim 2, characterized in that: The method of directly calculating the intra-frame target similarity using the current frame feature matrix includes: By calculating the similarity between the target feature and other target features, a similarity matrix is ​​formed; The first matching similarity matrix is ​​calculated by using the softmax operation on the similarity matrix.

5. The method according to claim 2, characterized in that: The similarity between objects is further measured by performing self-comparison learning on multiple frames, including: According to the circular correlation consistency, the forward correlation matrix and the reverse correlation matrix are calculated; A second matching similarity matrix is ​​calculated according to the forward correlation matrix and the reverse correlation matrix.

6. The method according to claim 1, characterized in that The method utilizes the consistency of cross-frame matching and continuous matching, adjusts the matching similarity, calculates the cross-matching loss, and extracts features that are beneficial to target association, including: Calculate the matching similarity matrix between different frames; According to the matching similarity matrix between different frames, the JS divergence between different frames is calculated; According to the JS divergence, the cross comparison loss between different frames is calculated.

7. The method according to claim 6, characterized in that The cross comparison loss between different frames is calculated according to the JS divergence, including: The cross-matching loss is calculated according to the following formula: Among them, S 1→3 represents the similarity matrix of the target in frame 1 matching frame 3, S 3→1 Represents the similarity matrix of the target in frame 3 matching frame 1, Represents the matching similarity matrix of the target in frame 1 matched to frame 3, Represents the matching similarity matrix of the target in frame 3 matched to frame 1.

8. The method according to claim 1, characterized in that The method of simultaneously considering the occluded, disappeared, and reappeared targets, re-matching these fuzzy targets, and calculating the fuzzy comparison loss to improve the certainty of subsequent target association includes: Find the blurred target in the first frame based on similarity; Find the blurred target in the second frame based on the similarity; The similarity of these fuzzy targets is calculated again to obtain a similarity matrix.

9. The method according to claim 8, characterized in that The similarity of these fuzzy targets is calculated again to obtain a similarity matrix, including: The similarity matrix is ​​calculated according to the following formula: in, and The similarity of these fuzzy targets is calculated again according to the formula to obtain the similarity matrix, N r and M r are the number of blurred objects in frame 1 and frame 2, respectively.

10. The method according to claim 1, characterized in that The comprehensive self-contrast loss, cross-contrast loss and fuzzy contrast loss are used to calculate the total loss, including: The total loss is calculated according to the following formula: L un_reid =L(I t ,I t-1 ,I t-2 )=L sc +L cc +L ac Among them, I t ,I t-1 ,I t-2 Represents three consecutive frames as input, L un_reid represents the total loss, L sc represents the self-alignment loss, L cc represents the cross-matching loss, L ac Represents the fuzzy comparison loss.

11. A joint detection and tracking device, characterized in that: The joint detection and tracking device comprises: The self-alignment module is used to train through direct intra-frame matching and indirect inter-frame matching, calculate the self-alignment loss, and obtain discriminative representations; The cross-matching module is used to utilize the consistency of cross-frame matching and continuous matching, calculate the cross-matching loss, adjust the matching similarity, and extract features that are beneficial to target association; The fuzzy comparison module is used to simultaneously consider occluded, disappeared, and reappearing targets, and to improve the certainty of subsequent target association by re-matching these fuzzy targets and calculating the fuzzy comparison loss; The loss calculation module is used to combine the self-contrast loss, cross-contrast loss and fuzzy contrast loss to calculate the total loss; The tracking and identification module is used to track and identify the target based on the total loss.

12. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; a computer program is stored in the memory, and when the computer program is executed by the processor, the processor executes the steps of any one of the methods described in claims 1 to 10.

13. A computer-readable storage medium, characterized in that: It stores a computer program executable by an electronic device, and when the computer program runs on the electronic device, the electronic device executes the steps of the method according to any one of claims 1 to 10.

Images