Detection model training method, tracking method, device, storage medium and product

Abstract

The application discloses a kind of detection model training method, tracking method, equipment, storage medium and product, the target detection model includes backbone network, detection module and ReID module, the backbone network is connected with detection module, ReID module, the detection module includes first neck network and detection head, the ReID module includes second neck network and ReID head, the second neck network is identical with the architecture of the first neck network;The training method includes freezing the ReID module, using detection dataset to supervise training the backbone network and detection module;Freeze the parameters of the backbone network and detection module, using video data to unsupervised training the ReID module.The application fully utilizes the detection result of detection module when unsupervised training ReID module, reduces the demand of artificial data labeling.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of computer vision, and particularly relates to a detection model training method, a tracking method, a device, a storage medium and a product. BACKGROUND

[0002] Multi-object tracking is an important technology in the field of computer vision, and its main purpose is to assign a unique ID number to the same target in a video sequence to generate a track. A powerful multi-object tracking solution can effectively cope with real-world challenges in various scenarios, including but not limited to security defense, traffic management, industrial production, intelligent monitoring and autonomous driving.

[0003] Currently, multi-object tracking algorithms are usually carried out under the Tracking-By-Detection (TBD) based tracking, which includes a detection stage and a data association stage. First, the video sequence is detected by a target detector to locate the target in the image; then, in the data association stage, the data association algorithm is used to associate the targets across frames to achieve the tracking task.

[0004] In the data association stage, TBD can be divided into two types according to the input data, as shown in Figure 1 and Figure 2 The type shown in Figure 1 only uses the detection result as the input of the data association stage, and the data association algorithm only uses the physical information of the detection result for association. Figure 2 The type shown in Figure 2 uses the detection result and the appearance feature as the input, and uses the detection result and the appearance feature for association, The type shown in

[0005] has better performance. Figure 2 Because the type shown in requires an appearance feature, the task of target detection and target corresponding appearance feature extraction needs to be completed in the detection stage. Therefore, the target detection algorithm can be divided into two types: Separate Detection and Embedding (SDE) and Joint Detection and Embedding (JDE). The SDE method first detects the target in the image through the detector, and then extracts the appearance feature of the target through the Re-identification (ReID) network. This method usually has high accuracy, but the two network inferences make the method rate lower. The JDE method directly completes the detection task and the appearance feature extraction task through a single network, which can reduce the network calculation cost, ensure the detection accuracy, and obtain faster inference speed.

[0006] JDE is a general term for a class of model structures used for joint object detection and re-identification tasks. JDE methods generally include two main branches, namely the detection branch and the ReID branch. The process of training JDE models relies on existing data, where the detection branch generally uses supervised learning methods guided by detection boxes with artificial annotations, and the ReID branch is supervised by the unique ID number of the target to help the model learn the unique features of each target. However, constructing a tracking dataset with ID numbers requires a large amount of manual annotation, which is costly and limits the widespread application of JDE methods. To reduce the cost of manual annotation, unsupervised learning ReID branch methods have emerged, which generally use well-annotated continuous video sequences as datasets, train by inputting two images of adjacent N frames, and process the output ReID feature vector through strategies, and optimize the ReID branch by calculating the loss. However, this method still has limitations, as it requires the dataset to be a manually annotated continuous video sequence, while existing detection datasets do not consider the continuity of video sequences and are mostly in the form of discrete images.

[0007] In the detection stage, existing JDE methods usually have a simple design for their ReID modules. Currently, some methods map the features output by the backbone network to Re-ID feature vectors using a single convolutional layer, or increase the number of channels and stack them on the detection output to generate Re-ID feature vectors. Although these methods are widely recognized for their simplicity and efficiency, they often affect the performance of the detection module and the ReID module in practical applications. Specifically, existing ReID module structures are simple and do not fully meet the needs of feature representation for the Re-ID task. Since the ReID module and the detection module share features from the backbone network, there is competition when accessing shared features, resulting in insufficient feature information for the Re-ID module and the detection module. Ultimately, this structural design deficiency leads to suboptimal performance for both detection and Re-ID tasks.

[0008] The availability of training data in the detection stage directly affects the scalability of the JDE method in different scenarios. Although discrete detection datasets have become increasingly available in the field of computer vision in recent years, these datasets cannot be directly used as input data for joint models, limiting their application in multi-target tracking. In existing ReID methods, supervised learning methods rely on accurate ID annotations in video sequences. However, such datasets are currently very limited in resources and require extensive manual annotation during production, resulting in high costs. Meanwhile, unsupervised ReID learning methods also face similar problems: although they do not rely on ID annotations, they still require extensive detection labeling of video sequences in specific scenarios. Therefore, how to effectively utilize existing discrete data resources and reduce the need for manual annotation is a major challenge in current technology.

[0009] In existing target tracking technologies, the data association stage faces an important problem: for long-lost trajectories, effective trajectory recovery is often difficult. The main reason for this problem is that when a target is lost for a long time, the prediction accuracy of the Kalman filter for target motion gradually decreases. After a long time of prediction by the motion model of the Kalman filter for a long-lost trajectory, the state information will have a large error, affecting the subsequent trajectory matching process. Existing data association algorithms, whether based on physical information IoU (Intersection over Union) matching or based on appearance information IoU matching, rely on the state information predicted by the Kalman filter for matching. However, in the case of a long-lost target, the accumulation of prediction errors makes the state prediction result of the Kalman filter no longer reliable, leading to the discarding of these long-lost trajectories in the data association process. Although this approach can temporarily avoid false matching, it also reduces the overall performance of the data association algorithm, especially in complex or long-time tracking scenarios. SUMMARY

[0010] The present application aims to provide a detection model training method, a tracking method, a device, a storage medium and a product to solve at least one of the problems of poor Re-ID task performance, large amount of manual data annotation and inability to effectively recover long-lost trajectories in the prior art.

[0011] The present application is to solve the above technical problems by the following technical solutions: a target detection model training method, the target detection model comprising a backbone network, a detection module and a ReID module, the backbone network being connected with the detection module and the ReID module, the detection module comprising a first neck network and a detection head, the ReID module comprising a second neck network and a ReID head, the second neck network having the same architecture as the first neck network; the training method comprising:

[0012] Freezing the ReID module, the backbone network and the detection module are supervised training by using the detection dataset; wherein, the detection dataset includes image data and its label;

[0013] Freezing the parameters of the backbone network and the detection module, the ReID module is unsupervised training by using the video data.

[0014] Further, the ReID module is unsupervised training by using the video data, comprising:

[0015] The backbone network extracts features from the continuous frame images in the video data to obtain feature maps of different scales;

[0016] The detection module detects the feature maps of different scales to obtain detection feature maps;

[0017] The ReID module re-identifies the feature maps of different scales to obtain ReID feature maps; wherein, each detection feature map corresponds to one ReID feature map;

[0018] According to the detection feature map, the ReID feature map corresponding to the detection feature map is screened to obtain ReID positive samples;

[0019] According to the ReID positive sample set obtained from the adjacent two images, the ReID loss is calculated, and the parameters of the ReID module are adjusted according to the ReID loss, so as to realize the unsupervised training of the ReID module.

[0020] Further, according to the detection feature map, the ReID feature map corresponding to the detection feature map is screened, comprising:

[0021] Determine whether the confidence of the detection feature map is greater than the first confidence threshold, if yes, retain the ReID feature map corresponding to the detection feature map; otherwise, eliminate the ReID feature map corresponding to the detection feature map;

[0022] The detection feature map corresponding to the retained ReID feature map is subjected to an overlapping elimination operation to obtain the ReID feature map corresponding to the retained detection feature map, that is, to obtain the ReID positive sample.

[0023] Further, the specific calculation formula of the ReID loss is:

[0024]

[0025] Wherein, L ID represents the ReID loss, L intra represents the internal loss, L inter represents the inter-frame loss, L cycledenotes the cycle constraint loss; N denotes the number of ReID positive samples obtained from the previous frame of the two adjacent frames of images, and M denotes the number of ReID positive samples obtained from the latter frame of the two adjacent frames of images; S ij denotes the similarity between the i-th target and the j-th target, and S ij is composed of a placeholder, and the similarity matrix S has N+M rows and N+M+1 columns; m denotes an interval value, j' denotes the subscript value of the second largest element in the i-th row of the similarity matrix S, j* denotes the subscript value of the largest element in the i-th row of the similarity matrix S, and argmax denotes finding j when S ij is the largest; e i , e j respectively denote the i-th and j-th ReID positive samples in the ReID positive sample set obtained from the two adjacent frames of images; e denotes the average value of all elements in the similarity matrix S except the diagonal elements, and || ||2 denotes the two-norm.

[0026] Based on the same concept, the application also provides a target tracking method, comprising:

[0027] calling a trained target detection model; wherein the target detection model is trained by using the target detection model training method described above;

[0028] detecting the current frame of image by using the trained target detection model to obtain the current frame detection result and its corresponding ReID feature map; wherein the current frame detection result comprises the current frame prediction box and its detection information;

[0029] tracking the target according to the historical trajectory, the current frame detection result and its corresponding ReID feature map.

[0030] Further, tracking the target according to the historical trajectory, the current frame detection result and its corresponding ReID feature map comprises:

[0031] judging whether the current prediction box belongs to a high-score box or a low-score box according to the detection information of the current frame prediction box;

[0032] performing one-time matching according to the historical trajectory, the high-score box and its corresponding ReID feature map, updating and outputting the trajectory of the target corresponding to the high-score box which succeeds in one-time matching;

[0033] for the historical trajectory which fails in one-time matching, when the missing time is less than a first time threshold, performing two-time matching according to the historical trajectory and the high-score box, updating and outputting the trajectory of the target corresponding to the high-score box which succeeds in two-time matching, and marking and outputting the target corresponding to the high-score box which fails in two-time matching as a new trajectory;

[0034] According to the unsuccessful history track and the low-score frame of the second matching, the third matching is performed, and for the target corresponding to the low-score frame of the third matching, the track is updated and output; for the low-score frame of the third matching, the track is discarded;

[0035] For the history track of the third matching unsuccessful and the history track of the first matching unsuccessful and the loss time greater than or equal to the first time threshold, when the loss time is greater than the second time threshold, the history track is discarded; when the loss time is less than or equal to the second time threshold, the history track is retained and output; wherein the second time threshold is greater than the first time threshold.

[0036] Further, according to the history track, the high-score frame and the ReID feature map corresponding thereto, the first matching is performed, comprising:

[0037] Step C1: initialize the track loss time counter t;

[0038] Step C2: judge whether the track loss time counter t is less than the second time threshold, if yes, go to step C3; if no, output the unmatched high-score frame and the unmatched history track;

[0039] Step C3: select the track with the loss time t from the history track, and perform the cosine similarity calculation on the track with the loss time t and the high-score frame and the ReID feature map corresponding thereto;

[0040] Step C4: according to the cosine similarity calculation result, the Hungarian algorithm is used to match the track with the loss time t and the target corresponding to the high-score frame, and for the target corresponding to the high-score frame matched successfully, the track is updated and output; for the high-score frame and the history track matched unsuccessfully, go to step C5;

[0041] Step C5: the track loss time counter t is incremented by 1, and go to step C2.

[0042] Based on the same concept, the present application also provides an electronic device, comprising a memory, a processor and a computer program / instruction stored in the memory, wherein the processor executes the computer program / instruction to realize the target detection model training method or the target tracking method as described above.

[0043] Based on the same concept, the present application also provides a computer readable storage medium, which stores a computer program / instruction, wherein the computer program / instruction is executed by a processor to realize the target detection model training method or the target tracking method as described above.

[0044] Based on the same concept, the present application also provides a computer program product, comprising a computer program / instruction, wherein the computer program / instruction is executed by a processor to realize the target detection model training method or the target tracking method as described above.

[0045] Advantages

[0046] Compared with the prior art, the advantages of the present application are:

[0047] The target detection model of the present application not only includes a detection module, but also includes a ReID module, so that the target detection task and the re-identification task are realized by a single target detection model; the target detection model of the present application effectively reduces the dependence of the re-identification task on the features of the backbone network by integrating the features of the backbone network at different scales, thereby realizing the excellent performance of the ReID task; when the ReID module is unsupervisedly trained, the detection results of the detection module are fully utilized, and the demand for manual data labeling is reduced; compared with the traditional method, the training method of the present application is more economical and efficient, and the scalability of the JDE model in different scenarios is improved.

[0048] The tracking method of the present application effectively recovers long-term lost trajectories by setting a first time threshold and a second time threshold and adopting different matching strategies for trajectories with different loss times. BRIEF DESCRIPTION OF DRAWINGS

[0049] In order to more clearly illustrate the technical solutions of the present application, the drawings needed in the embodiment description will be briefly introduced as follows: obviously, the drawings in the following description are only one embodiment of the present application, and other drawings can be obtained by those skilled in the art without creative labor on the basis of these drawings.

[0050] Figure 1 is an example diagram in the background art of the present application, in which the detection result is used as the input of the tracker;

[0051] Figure 2 is an example diagram in the background art of the present application, in which the detection result and the appearance feature are used as the input of the tracker;

[0052] Figure 3 is the architecture diagram of the target detection model in the embodiment of the present application;

[0053] Figure 4 is the flowchart of the training method of the target detection model in the embodiment of the present application;

[0054] Figure 5 is the flowchart of the unsupervised training of the ReID module in the embodiment of the present application;

[0055] Figure 6 is the flowchart of the target tracking in the embodiment of the present application;

[0056] Figure 7 is the flowchart of one-time matching in the embodiment of the present application. DETAILED DESCRIPTION

[0057] The technical solutions in the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative work belong to the scope of protection of the present application.

[0058] The technical solutions of the present application will be described in detail below with specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in some embodiments.

[0059] Embodiment one

[0060] Figure 3 The architecture diagram of the target detection model is shown, where C represents concatenation, U represents up-sampling, RepBlock is a neural network module composed of multiple convolution layers, and Conv represents a convolution layer. As shown in Figure 3 The target detection model includes a backbone network, a detection module and a ReID module, the backbone network is connected with the detection module and the ReID module, the detection module includes a first neck network and a detection head, the ReID module includes a second neck network and a ReID head, the first neck network and the second neck network adopt the same architecture but different parameters. The backbone network is used for feature extraction of the input image to obtain feature maps of different scales; the detection module is used for detection of the feature maps of different scales to obtain target detection results at different scales; the ReID module is used for re-identification of the feature maps of different scales to obtain ReID feature maps at different scales, i.e. the appearance features of the target at different scales.

[0061] In this embodiment, the backbone network adopts EfficientNet; the first neck network and the second neck network both adopt the neck network in the Yolov6 model, and the ReID head adopts a single-layer convolution and is used for channel transformation. The structure flexibility of EfficientNet is outstanding, which can adopt a multi-branch architecture, including a 3x3 convolution branch, a 1x1 convolution branch and an identity mapping branch. This multi-branch design enables the EfficientNet network to effectively integrate different features and receptive fields, thereby ensuring higher training accuracy than single-branch network structures, especially in complex training scenarios and tasks. This flexibility not only improves the training effect, but also helps to maintain the high quality of the ReID feature map during the detection process. The backbone network includes three output layers, which are represented by the three trapezoids in Figure 3 , the closer to the end of the input image, the larger the size of the feature map output. Let the input image be 3xW IN xH IN, the feature map from the first output layer is 128 x (W IN / 8) x (H IN / 8), the feature map from the second output layer is 256 x (W IN / 16) x (H IN / 16), and the feature map from the third output layer is 512 x (W IN / 32) x (H IN / 32), where W IN denotes the width of the input image, and H IN denotes the height of the input image.

[0062] The detection head includes a position regression head, a confidence head, and a classification head. The position regression head is used to predict the position and size of the target. Due to the size difference of the target, the detection module detects targets of different scales by outputting feature maps of different scales. Smaller scale feature maps contain more semantic information and are suitable for detecting larger targets. During the training process, the loss of the position regression head is calculated by the SIoU (Scale Intersection over Union) function to optimize the position regression accuracy. The SIoU loss includes angle loss, distance loss, shape loss, and intersection over union loss, where the calculation formula of the angle loss is:

[0063]

[0064] where Λ denotes the angle loss, σ denotes the distance between the real box and the predicted box, c h denotes the height of the minimum bounding rectangle of the real box and the predicted box, denotes the center coordinates of the real box, denotes the center coordinates of the predicted box.

[0065] The calculation formula of the distance loss is:

[0066]

[0067] γ = 2 - Λ (6)

[0068] where Δ denotes the distance loss, γ, ρ t denote intermediate quantities, c w denotes the width of the minimum bounding rectangle of the real box and the predicted box.

[0069] The calculation formula of the shape loss is:

[0070]

[0071] where Ω denotes the shape loss, ω w , ω h denote intermediate quantities, w gt and h gtrespectively, w and h represent the width and height of the prediction box respectively, and θ represents a shape cost control coefficient, and its recommended value is 4.

[0072] The calculation formula of the intersection over union loss is:

[0073]

[0074] wherein, IoU represents the intersection over union loss, B GT represents the real box, and B represents the prediction box.

[0075] The calculation formula of the SIoU loss is:

[0076]

[0077] wherein, L SloU represents the SIoU loss.

[0078] The confidence head is used to predict the existence confidence of the target, that is, the probability of the target in the detection box, and the output value is mapped to a probability distribution of 0-1. In the training process, the loss L cof of the confidence is calculated by comparing with the real label using the binary cross entropy loss function, and then guiding the optimization of the confidence. The target that does not meet the confidence condition is filtered through the confidence threshold, and the calculation formula of the confidence loss is:

[0079]

[0080] wherein, K represents the number of samples, y k represents the real label (0 or 1) of the kth sample, p k represents the prediction probability of the kth sample when the real label is 1.

[0081] The classification head is used to assign accurate classification labels to the targets in the prediction box. In the output layer, the classification head maps the features to the class space through a fully connected layer, and outputs the probability value of each class, so as to determine the most possible class corresponding to the target. In the training process, the classification loss is calculated by the VariFocal function, which can balance the class imbalance problem and optimize the classification accuracy. The specific calculation formula of the classification loss is:

[0082]

[0083] wherein, L cls represents the classification loss; represents the prediction probability; y represents the real label (0 or 1); α represents an adjustment factor, which is used to control the weight of positive and negative samples; and ε represents a focusing parameter, which is used to adjust the influence of the sample difficult to classify.

[0084] The target detection model of the present application increases a ReID module on the basis of the original model, the second neck network has the same architecture as the first neck network and adopts the neck network of Yolov6, has the characteristics of strong scale feature fusion; the detection head and the ReID head use different convolution layers to output feature maps with different channel numbers to complete the detection task and the ReID task. The present application completes the detection task and the ReID task through a single target detection model, the detection module can efficiently and accurately detect targets, provide high-quality position information, confidence evaluation and classification labels, so as to realize more accurate multi-target tracking. The second neck network of the ReID module fuses and transforms the feature maps of different scales output by the backbone network, so as to extract the appearance features of the target under different scales, and these fused features are then transmitted to the ReID head through three different levels, for obtaining the appearance representation of the target under different scales, to support more accurate target re-identification. Through this design, the ReID module can make full use of multi-scale features, so as to improve the re-identification performance of the target under different perspectives and scales, and further enhance the stability and accuracy of target tracking.

[0085] Figure 4 The training method of the target detection model provided by the present application is shown. As shown in Figure 4 , the training method comprises the following steps:

[0086] Step A1: freeze the ReID module, and use the detection data set to supervise the training of the backbone network and the detection module.

[0087] The detection data set includes image data and its label, and the labeled image data is used to supervise the training of the backbone network and the detection module until the backbone network and the detection module converge. In the supervised training process, the ReID module does not participate in the training, and the image data in the detection data set is discontinuous.

[0088] Step A2: freeze the parameters of the backbone network and the detection module, and use the video data to unsupervisedly train the ReID module.

[0089] Traditional ReID training is usually carried out in a supervised manner, which requires the data set to have the ID number of the target, and this manner brings huge manual cost. The application adopts an unsupervised manner to train the ReID module, the core idea of which relies on the space and time information to infer the similarity between targets, and uses strong and weak supervision signals to guide the training of the ReID module, so that the ReID training can be carried out in an unknown scene based on existing detection data. In the unsupervised training process of the ReID module, the backbone network and the detection module do not participate in the training, the data set is switched to continuous video data, the video data does not need to have any annotation information, but the images in the video data must be continuous, and the detection result of the detection module will guide the training of the ReID module. The ReID module outputs the ReID feature map corresponding to the detection result output by all detection heads, but only the ReID feature map selected according to the detection result of the detection module is the positive sample required by the ReID module.

[0090] In the specific embodiments of the application, as shown in Figure 5 the video data is used to unsupervisedly train the ReID module, which includes:

[0091] Step A2.1: The backbone network extracts features from the continuous frame images in the video data to obtain feature maps of different scales;

[0092] Step A2.2: The detection module detects the feature maps of different scales to obtain detection feature maps;

[0093] Step A2.3: The ReID module performs re-identification on the feature maps of different scales to obtain ReID feature maps; the detection feature maps and the ReID feature maps are only different in the number of channels and are the same in scale, and each detection feature map corresponds to one ReID feature map;

[0094] Step A2.4: According to the detection feature map, the ReID feature map corresponding to the detection feature map is screened to obtain ReID positive samples;

[0095] Step A2.5: Calculate the ReID loss according to the ReID positive sample set obtained from the adjacent two frame images, and adjust the parameters of the ReID module according to the ReID loss, to realize the unsupervised training of the ReID module.

[0096] In Step A2.4, the ReID feature maps corresponding to the detection feature maps are not all positive samples, and need to be screened. In the specific embodiments of the present application, the screening is performed through post-processing operations of the confidence of the detection feature map and a non-maximum suppression algorithm. Specifically, it is judged whether the confidence of each detection feature map is greater than a first confidence threshold, if yes, the ReID feature map corresponding to the detection feature map is retained; otherwise, the ReID feature map corresponding to the detection feature map is removed; the non-maximum suppression algorithm is used to perform an overlapping removal operation on the detection feature map corresponding to the retained ReID feature map, to obtain the ReID feature map corresponding to the retained detection feature map, i.e. to obtain the ReID positive sample.

[0097] In the screening process, only the detection results (i.e. the detection feature maps) of the detection module are used, and the ReID positive samples are screened along with the detection results.

[0098] The video data is input into the target detection model, and the ReID positive sample set obtained from the front and rear two images is represented as E∈{e0,e1,…,e N-1 N ,…,e N+M-1}, wherein N and M respectively represent the number of ReID positive samples in the front and rear images. The similarity matrix is defined as S∈R (N+M)×(N+M+1) The similarity matrix S in the present embodiment is a cosine similarity matrix, and can be represented as The similarity matrix S has N+M rows and N+M+1 columns, the N+M rows are the sum of the number of targets in the front and rear two images, and the N+M+1 columns are the sum of the number of targets in the front and rear two images plus a column of placeholders. Wherein S1∈R (0,N)×(0,N) and S4∈R (N,N+M)×(N,N+M) respectively represent the cosine similarity between the N targets in the front image and the cosine similarity between the M targets in the rear image, and the diagonal elements are set to negative infinity, because they are the cosine similarity calculated for the same target; S3∈R (0,N)×(N,N+M) and S2∈R (N,N+M)×(0,N) respectively represent the cosine similarity between the targets in the front and rear two images; S5∈R (0,N+M)×(N+M,N+M+1) represents a placeholder, i.e. represents that there is no matching target between the front and rear two images. The specific calculation formula of each element in the cosine similarity matrix S is:

[0099]

[0100] Wherein S ij represents the similarity between the i th target and the j th target, e i , e j ​respectively represent the i-th and j-th ReID positive sample of the ReID positive sample set E obtained from the front and rear two frames of images, represents the average value of all elements in the similarity matrix except the diagonal elements, and |||2 represents the two norm.

[0101] In step A2.5, the ReID loss is calculated by the intra-image loss, the inter-frame loss and the cycle constraint loss. The intra-image loss L intra guided by the strong supervision signal, L intra represents that there will be no same target in the same image, so the cosine similarity between the intra-image targets should tend to 0, and the specific calculation formula is:

[0102]

[0103] The inter-frame loss L inter guided by the weak supervision signal, L inter represents that most of the targets in the adjacent two frames of images can be matched two by two, and there will be an S ij than other cosine similarities. That is, for a target at the previous moment, it can find a target with the largest similarity at the next moment, and the cosine similarity between them should be larger than other cosine similarities, and the unmatched target will be trained to a placeholder. The specific calculation formula of the inter-frame loss is:

[0104]

[0105] wherein m represents an interval value (usually taking a default value of 0.5), represents the minimum distance of the cosine similarity between different targets; j' represents the subscript value of the second largest element in the i-th row of the similarity matrix, j* represents the subscript value of the largest element in the i-th row of the similarity matrix, and argmax represents finding the j of the maximum S ij .

[0106] The cycle constraint loss L cycle is used to indicate that the cosine similarity between any two objects will not change due to its position in vector multiplication, and the specific calculation formula is:

[0107]

[0108] Therefore, the calculation formula of the ReID loss is:

[0109]

[0110] The backbone network and the detection module in the target detection model are trained by using existing detection data sets, and the ReID module is unsupervised trained under the guidance of the detection result output by the detection module by using video data, which overcomes the limitation of the continuity requirement of the traditional method on the video sequence, effectively reduces the dependence on manual labeling in the training process, and realizes the goal of training an efficient JDE model on discrete detection data sets.

[0111] Embodiment two

[0112] The target tracking method provided by the embodiment of the application comprises the following steps:

[0113] Step B1: obtaining a current frame detection result and a corresponding ReID feature map.

[0114] The current frame image is detected to obtain the current frame detection result and the corresponding ReID feature map, wherein the current frame detection result comprises a current frame prediction box and detection information thereof, and the detection information comprises the confidence of the current frame prediction box. The current frame image is detected by using a target detection model, and the target detection model can be an existing model or a target detection model trained by the training method of the embodiment one of the application.

[0115] Step B2: performing target tracking according to the historical trajectory, the current frame detection result and the corresponding ReID feature map.

[0116] The target tracking is performed according to the historical trajectory, the current frame detection result and the corresponding ReID feature map, and the mutual correlation or matching between the historical trajectory and the current frame detection target is performed, so that the same ID number is assigned to the cross-frame target, and multi-target tracking is realized.

[0117] In the specific embodiments of the application, as shown in Figure 6 The target tracking according to the historical trajectory, the current frame detection result and the corresponding ReID feature map comprises:

[0118] Step B2.1: determining whether the current prediction box belongs to a high-score box or a low-score box according to the detection information of the current frame prediction box.

[0119] When the confidence of the current frame prediction box is greater than a second confidence threshold, the current frame prediction box is determined to be a high-score box; and when the confidence of the current frame prediction box is less than or equal to the second confidence threshold, the current frame prediction box is determined to be a low-score box. The high-score box and the low-score box are determined for all current frame prediction boxes, and thus a high-score box set and a low-score box set are obtained.

[0120] Step B2.2: According to the historical trajectory, the high-score bounding box and its corresponding ReID feature map, once matching is performed, for the target corresponding to the high-score bounding box that once matching is successful, the trajectory is updated and output; for the historical trajectory and the high-score bounding box that once matching is unsuccessful, step B2.3 is entered;

[0121] Step B2.3: For the historical trajectory that once matching is unsuccessful, it is judged whether the loss time of the historical trajectory is less than the first time threshold T short , if yes, step B2.4 is entered; otherwise, step B2.6 is entered;

[0122] Step B2.4: According to the historical trajectory and the high-score bounding box, twice matching is performed, for the target corresponding to the high-score bounding box that twice matching is successful, the trajectory is updated and output; for the target corresponding to the high-score bounding box that twice matching is unsuccessful, it is marked as a new trajectory and output; for the historical trajectory that twice matching is unsuccessful, step B2.5 is entered;

[0123] Step B2.5: According to the historical trajectory that twice matching is unsuccessful and the low-score bounding box, thrice matching is performed, for the target corresponding to the low-score bounding box that thrice matching is successful, the trajectory is updated and output; for the low-score bounding box that thrice matching is unsuccessful, it is discarded; for the historical trajectory that thrice matching is unsuccessful, step B2.6 is entered;

[0124] Step B2.6: For the historical trajectory that thrice matching is unsuccessful and the historical trajectory that once matching is unsuccessful and the loss time is greater than or equal to the first time threshold, it is judged whether the loss time of the historical trajectory is greater than the second time threshold T long , if yes, step B2.7 is entered; otherwise, step B2.8 is entered;

[0125] Step B2.7: The historical trajectory is discarded;

[0126] Step B2.8: The historical trajectory is retained and output.

[0127] The first time threshold and the second time threshold are used to determine the matching stage in which the historical trajectory participates, the second time threshold T long is greater than the first time threshold T short , the first time threshold T short and the second time threshold T long are in units of image frames, the first time threshold of the embodiment is set to 30 frames, and the second time threshold can be dynamically adjusted according to the tracking scene, if the loss time of the target in the scene is too long, the second time threshold is set to be larger, and the second time threshold of the embodiment is set to 200 frames.

[0128] After the steps B2.1 to B2.8 are performed, one round of matching is completed, and each round of matching includes three times of matching, i.e., high-score ReID cascade matching (corresponding to step B2.2), high-score IoU matching stage (corresponding to step B2.4), and low-score matching stage (corresponding to step B2.5), and all the output trajectories are used as historical trajectories for the next round of matching. The task of a single round of matching is to assign the detected target in the current frame to the historical trajectory and update the historical trajectory. The historical trajectory is a target trajectory formed before the current frame and the corresponding ReID feature map. The loss time of a certain trajectory is recorded from the moment when the trajectory fails to match any detected target in the single round of matching, and if the subsequent matching still fails, the loss time is increased by 1 after each round of matching. If the matching is successful again, the loss time is reset to 0.

[0129] In the specific embodiments of the present application, as shown in Figure 7 According to the historical trajectory, the high-score bounding box and the corresponding ReID feature map, one matching is performed, which includes:

[0130] Step C1: initialize the trajectory loss time counter t, and the initial value of the trajectory loss time counter i is 0;

[0131] Step C2: determine whether the trajectory loss time counter t is less than the second time threshold, if yes, go to step C3; if no, output the unmatched high-score bounding box and the unmatched historical trajectory, and go to step B2.3;

[0132] Step C3: select the trajectory with the loss time t from the historical trajectory, and perform cosine similarity calculation on the trajectory with the loss time t and the high-score bounding box and the corresponding ReID feature map;

[0133] Step C4: according to the cosine similarity calculation result, the Hungarian algorithm is used to match the trajectory with the loss time t and the target corresponding to the high-score bounding box, i.e., the trajectory of the target corresponding to the high-score bounding box which is successfully matched is updated and output; the high-score bounding box and the historical trajectory which are not successfully matched are transferred to step C5;

[0134] Step C5: the trajectory loss time counter t is increased by 1, and step C2 is transferred.

[0135] The present application can be applied to vehicle target detection and tracking, and when the target detection model is trained, the images in the detection data set are images containing vehicles, and the video data is a vehicle motion video.

[0136] The present application sets the first time threshold and the second time threshold, uses different matching strategies for trajectories with different loss times, effectively recovers long-term lost trajectories, improves the stability and accuracy of target tracking, reduces the adverse effects caused by the prediction error of the Kalman filter, and thus improves the accuracy of target re-identification and matching.

[0137] Embodiment Three

[0138] The embodiments of the present application also provide an electronic device, which comprises a memory, a processor and computer programs / instructions stored in the memory, and the processor executes the computer programs / instructions to implement the target detection model training method or the target tracking method in the embodiments of the present application.

[0139] Although not shown, the electronic device comprises a processor, which can perform various appropriate operations and processes according to programs and / or data stored in a read-only memory (ROM) or loaded from a storage section into a random access memory (RAM). The processor can be a multi-core processor or can include multiple processors. In some embodiments, the processor can include a general-purpose main processor and one or more special-purpose coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processing unit (NPU), a digital signal processor (DSP), etc. In the RAM, various programs and data required for device operation are also stored. The processor, the ROM, and the RAM are connected to each other through a bus. An input / output (I / O) interface is also connected to the bus.

[0140] The processor and the memory are used together to execute programs / instructions stored in the memory, and the programs / instructions are executed by a computer to implement the methods, steps or functions described in the above embodiments.

[0141] Although not shown, the embodiments of the present application also provide a computer readable storage medium having computer programs / instructions stored thereon, and the computer programs / instructions are executed by a processor to implement the target detection model training method or the target tracking method in the embodiments of the present application.

[0142] The readable storage medium includes permanent and non-permanent, movable and non-movable media, which can realize information storage by any method or technology. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device. According to the definition herein, the computer readable medium does not include transitory media such as modulated data signals and carriers.

[0143] Although not shown, the embodiments of the present application also provide a computer program product, comprising: computer programs / instructions, which, when executed by a processor, realize the target detection model training method or the target tracking method in the embodiments of the present application.

[0144] The above only discloses specific embodiments of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of changes or modifications within the technical scope disclosed by the present application, which should be covered within the protection scope of the present application.

Claims

1. A method for training an object detection model, characterized in that, The target detection model includes a backbone network, a detection module, and a ReID module. The backbone network is connected to the detection module and the ReID module. The detection module includes a first neck network and a detection head. The ReID module includes a second neck network and a ReID head. The second neck network has the same architecture as the first neck network. The training method includes: The ReID module is frozen, and the backbone network and detection module are trained in a supervised manner using the detection dataset; wherein, the detection dataset includes image data and its labels; The parameters of the backbone network and detection module are frozen, and the ReID module is trained unsupervised using video data; Unsupervised training of the ReID module using video data includes: The backbone network extracts features from consecutive frames of video data to obtain feature maps at different scales. The detection module detects feature maps at different scales to obtain detection feature maps; The ReID module re-identifies feature maps at different scales to obtain ReID feature maps; wherein, each detected feature map corresponds to one ReID feature map; Based on the detected feature map, the ReID feature maps corresponding to the detected feature map are filtered to obtain positive ReID samples; The ReID loss is calculated based on the positive ReID sample set obtained from two adjacent frames of images, and the parameters of the ReID module are adjusted according to the ReID loss to achieve unsupervised training of the ReID module.

2. The target detection model training method according to claim 1, characterized in that, Based on the detected feature map, the ReID feature map corresponding to the detected feature map is filtered, including: Determine whether the confidence level of the detected feature map is greater than a first confidence threshold. If so, retain the ReID feature map corresponding to the detected feature map; otherwise, discard the ReID feature map corresponding to the detected feature map. The overlap removal operation is performed on the detection feature map corresponding to the retained ReID feature map to obtain the ReID feature map corresponding to the retained detection feature map, which is the ReID positive sample.

3. The target detection model training method according to claim 1 or 2, characterized in that, The specific formula for calculating the ReID loss is as follows: Among them, L ID L represents the ReID loss. intra L represents internal loss. inter L represents the inter-frame loss. cycle denoted by cyclic constraint loss; N represents the number of positive ReID samples obtained from the previous frame of two adjacent frames, and M represents the number of positive ReID samples obtained from the next frame of two adjacent frames; S ij The similarity between the i-th target and the j-th target is represented by S, which is the sum of all similarity values. ij Let S be a similarity matrix consisting of a column of placeholders and a column of placeholders. The similarity matrix S has N+M rows and N+M+1 columns; m represents the interval value, j' represents the index of the second largest element in the i-th row of the similarity matrix S, j* represents the index of the largest element in the i-th row of the similarity matrix S, and argmax represents finding the maximum value in S. ij j;e at maximum i e j These represent the i-th and j-th positive ReID samples obtained from two adjacent frames of images, respectively. Let |||S| represent the average of all elements in the similarity matrix S excluding the diagonal elements, and |||2| represents the L2 norm.

4. A target tracking method, characterized in that, The tracking method includes: The pre-trained object detection model is invoked; wherein the object detection model is trained using the object detection model training method as described in any one of claims 1 to 3; The trained object detection model is used to detect the current frame image, and the detection result of the current frame and its corresponding ReID feature map are obtained; the detection result of the current frame includes the predicted bounding box of the current frame and its detection information; Target tracking is performed based on historical trajectories, current frame detection results, and their corresponding ReID feature maps.

5. The target tracking method according to claim 4, characterized in that, Target tracking is performed based on historical trajectories, current frame detection results, and their corresponding ReID feature maps, including: Based on the detection information of the predicted bounding box in the current frame, determine whether the current predicted bounding box is a high-resolution bounding box or a low-resolution bounding box; A matching is performed based on the historical trajectory, high-scoring bounding box and its corresponding ReID feature map. For the target corresponding to the high-scoring bounding box that is successfully matched, its trajectory is updated and output. For historical trajectories that failed to match once, if the loss time is less than the first time threshold, a second matching is performed based on the historical trajectory and the high-resolution bounding box. For targets corresponding to high-resolution bounding boxes that successfully match once, their trajectories are updated and output. For targets corresponding to high-resolution bounding boxes that failed to match once, they are marked as new trajectories and output. Based on the historical trajectories and low-scoring bounding boxes that failed the second-order matching, a third-order matching is performed. For targets corresponding to low-scoring bounding boxes that succeeded in the third-order matching, their trajectories are updated and output; for low-scoring bounding boxes that failed in the third-order matching, they are discarded. For historical trajectories that failed to match three times and those that failed to match once and whose loss time is greater than or equal to the first time threshold, if the loss time is greater than the second time threshold, the historical trajectory is discarded; if the loss time is less than or equal to the second time threshold, the historical trajectory is retained and output; where the second time threshold is greater than the first time threshold.

6. The target tracking method according to claim 5, characterized in that, A matching process is performed based on historical trajectories, high-resolution bounding boxes, and their corresponding ReID feature maps, including: Step C1: Initialize the trajectory loss time counter t; Step C2: Determine if the trajectory loss time counter t is less than the second time threshold. If yes, proceed to step C3; otherwise, output the unmatched high-scoring bounding boxes and the unmatched historical trajectories. Step C3: Select the trajectory with a loss time of t from the historical trajectories, and calculate the cosine similarity between the trajectory with a loss time of t and the high-resolution bounding box and its corresponding ReID feature map; Step C4: Based on the cosine similarity calculation results, the Hungarian algorithm is used to match the trajectory with a loss time of t with the target corresponding to the high-scoring box. For the target corresponding to the high-scoring box that is successfully matched, its trajectory is updated and output; for the high-scoring box and historical trajectory that are not successfully matched, proceed to step C5. Step C5: Increment the trajectory loss time counter t by 1, and proceed to step C2.

7. An electronic device comprising a memory, a processor, and a computer program / instructions stored in the memory, characterized in that, The processor executes the computer program / instructions to implement the target detection model training method as described in any one of claims 1 to 3 or the target tracking method as described in any one of claims 4 to 6.

8. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When the computer program / instructions are executed by the processor, they implement the target detection model training method as described in any one of claims 1 to 3 or the target tracking method as described in any one of claims 4 to 6.

9. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the target detection model training method as described in any one of claims 1 to 3 or the target tracking method as described in any one of claims 4 to 6.

Images