Detection model training method, tracking method, equipment, storage medium and product

Abstract

The invention discloses a detection model training method, a tracking method, equipment, a storage medium and a product, a target detection model comprises a backbone network, a detection module and a ReID module, the backbone network is connected with the detection module and the ReID module, the detection module comprises a first neck network and a detection head, the ReID module comprises a second neck network and a ReID head, and the detection head is connected with the detection module and the ReID module. The architecture of the second neck network is the same as that of the first neck network; the training method comprises the following steps: freezing the ReID module, and performing supervised training on the backbone network and the detection module by using a detection data set; and freezing parameters of the backbone network and the detection module, and performing unsupervised training on the ReID module by using video data. According to the method, when the ReID module is subjected to unsupervised training, the detection result of the detection module is fully utilized, and the requirement of manual data annotation is reduced.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a detection model training method, tracking method, device, storage medium and product. Background Art

[0002] Multi-target tracking is an important technology in the field of computer vision. Its main purpose is to assign unique ID numbers to the same targets in a video sequence to generate trajectories. A powerful multi-target tracking solution can effectively cope with real-world challenges in various scenarios, including but not limited to security defense, traffic management, industrial production, intelligent monitoring and autonomous driving.

[0003] At present, multi-target tracking algorithms are usually performed based on tracking-by-detection (TBD), which includes a detection phase and a data association phase. First, the video sequence is detected by a target detector to locate the target in the image; then, in the data association phase, these targets are associated across frames through a data association algorithm to achieve the tracking task.

[0004] In the data association stage, TBD can be divided into two types according to the different data inputs, such as Figure 1 and Figure 2 shown. Figure 1 The type shown uses only the detection results as input to the data association stage, and the data association algorithm only uses the physical information of the detection results for association. Figure 2 The type shown takes the detection results and appearance features as input, and uses the detection results and appearance features for association. Figure 2 The types shown tend to have better performance.

[0005] because Figure 2 The type shown has the requirement of appearance features, so in the detection stage, the tasks of target detection and extraction of appearance features corresponding to the target must be completed. Therefore, target detection algorithms can be divided into two types: Separate Detection and Embedding (SDE) and Joint Detection and Embedding (JDE). The SDE method first detects the target in the image through the detector, and then extracts the appearance features of the target through the Re-ID network (Re-identification, ReID). This method usually has higher accuracy, but the two network reasoning makes the method rate lower. The JDE method directly completes the detection task and appearance feature extraction task through a single network, which can reduce the network calculation cost and obtain faster reasoning speed while ensuring detection accuracy.

[0006] JDE is a general term for a class of model structures used for joint object detection and re-identification tasks. JDE methods usually include two main branches, namely the detection branch and the ReID branch. The process of training the JDE model relies on existing data. The detection branch generally adopts a supervised learning method to guide the training through manually annotated detection boxes; while the ReID branch is supervised by the unique ID number of the target to help the model learn the unique characteristics of each target. However, building a tracking dataset with ID numbers requires a lot of manual annotation, which is costly and limits the widespread application of the JDE method. In order to reduce the cost of manual annotation, unsupervised learning methods for the ReID branch have emerged. Such methods usually use well-annotated continuous video sequences as datasets, train by inputting two images of adjacent N frames, and perform strategic processing on the output ReID feature vectors, and optimize the ReID branch by calculating the loss. However, this method still has limitations. Its dataset must be a manually annotated continuous video sequence, while the existing detection datasets usually do not consider the continuity of video sequences and are more in the form of discrete images.

[0007] In the detection phase, existing JDEs usually have a simple design for their ReID modules. Currently, some methods generate Re-ID feature vectors by mapping the features output by the backbone network into Re-ID feature vectors using a single convolutional layer, or by increasing the number of channels and superimposing them on the detection output. Although these methods are widely recognized for their simple structure and high efficiency, they often affect the performance of the detection module and the Re-ID module in practical applications. Specifically, the existing Re-ID module has a simple structure and fails to fully meet the requirements of the Re-ID task for feature representation. Since the Re-ID module and the detection module share feature information from the backbone network, there is competition between the two when accessing the shared features, resulting in insufficient feature information obtained by the Re-ID module and the detection module. Ultimately, this structural design deficiency leads to suboptimal performance of both the detection and Re-ID tasks.

[0008] The availability of training data in the detection phase directly affects the scalability of the JDE method in different scenarios. Although discrete detection datasets have become increasingly accessible in the field of computer vision in recent years, these datasets cannot be directly used as input data for joint models, which limits their application in multi-target tracking. Among the existing ReID methods, supervised learning methods rely on accurate ID annotations in video sequences. However, the resources of such datasets are currently very limited, and the production process requires a large amount of manual annotation, resulting in high costs. At the same time, unsupervised ReID learning methods also face similar problems: although it does not rely on ID annotation, it still requires a large amount of detection labeling for video sequences of specific scenes. Therefore, how to effectively utilize existing discrete data resources and reduce the need for manual annotation is a major challenge in current technology.

[0009] In the existing target tracking technology, the data association stage faces an important problem, that is, it is often impossible to effectively recover the trajectory for a long time that has been lost. The main reason for this problem is that when the target is lost for too long, the prediction accuracy of the Kalman filter for the target motion gradually decreases. After the long-lost trajectory is predicted by the motion model of the Kalman filter for a long time, the state information will have a large error, which will affect the subsequent trajectory matching process. The existing data association algorithm, whether it is IoU (Intersection over Union) matching based on physical information or IoU matching based on appearance information, relies on the state information predicted by the Kalman filter for matching. However, when the target is lost for too long, the accumulation of prediction errors makes the state prediction results of the Kalman filter no longer reliable, resulting in the abandonment of these long-lost trajectories in the data association process. Although this approach can temporarily avoid false matching, it also reduces the overall performance of the data association algorithm, especially in complex or long-term tracking scenarios. Summary of the invention

[0010] The purpose of the present invention is to provide a detection model training method, tracking method, device, storage medium and product to solve at least one of the problems in the prior art of poor Re-ID task performance, large amount of manual data annotation and inability to effectively recover trajectories that have been lost for a long time.

[0011] The present invention solves the above technical problems through the following technical solutions: a target detection model training method, the target detection model includes a backbone network, a detection module and a ReID module, the backbone network is connected to the detection module and the ReID module, the detection module includes a first neck network and a detection head, the ReID module includes a second neck network and a ReID head, the second neck network has the same architecture as the first neck network; the training method includes:

[0012] Freeze the ReID module, and use the detection data set to perform supervised training on the backbone network and the detection module; wherein the detection data set includes image data and its labels;

[0013] The parameters of the backbone network and the detection module are frozen, and the ReID module is trained unsupervisedly using video data.

[0014] Furthermore, the ReID module is unsupervisedly trained using video data, including:

[0015] The backbone network extracts features from continuous frame images in the video data to obtain feature maps of different scales;

[0016] The detection module detects feature maps of different scales to obtain detection feature maps;

[0017] The ReID module re-identifies feature maps of different scales to obtain ReID feature maps; wherein each detection feature map corresponds to a ReID feature map;

[0018] According to the detection feature graph, a ReID feature graph corresponding to the detection feature graph is screened to obtain a ReID positive sample;

[0019] The ReID loss is calculated according to the ReID positive sample set obtained from two adjacent frames of images, and the parameters of the ReID module are adjusted according to the ReID loss to achieve unsupervised training of the ReID module.

[0020] Further, screening the ReID feature graph corresponding to the detection feature graph according to the detection feature graph includes:

[0021] Determine whether the confidence of the detection feature map is greater than a first confidence threshold, if so, retain the ReID feature map corresponding to the detection feature map; otherwise, remove the ReID feature map corresponding to the detection feature map;

[0022] An overlapping elimination operation is performed on the detection feature map corresponding to the retained ReID feature map to obtain the ReID feature map corresponding to the retained detection feature map, that is, the ReID positive sample is obtained.

[0023] Furthermore, the specific calculation formula of the ReID loss is:

[0024]

[0025] Among them, L ID represents the ReID loss, L intra Represents internal loss, L inter Indicates the inter-frame loss, L cyclerepresents the cycle constraint loss; N represents the number of positive ReID samples obtained from the previous frame of two adjacent frames, and M represents the number of positive ReID samples obtained from the next frame of two adjacent frames; S ij Represents the similarity between the i-th target and the j-th target, which is composed of all similarities S ij and a column of placeholders to form a similarity matrix S. The similarity matrix S has N+M rows and N+M+1 columns. m represents the interval value, j' represents the subscript value of the second largest element in the i-th row of the similarity matrix S, j* represents the subscript value of the largest element in the i-th row of the similarity matrix S, and argmax represents finding the value of S. ij j at maximum; e i 、e j They represent the i-th and j-th ReID positive samples in the ReID positive sample set obtained from two adjacent frames of images respectively; e represents the average value of all elements in the similarity matrix S except the diagonal elements, and || ||2 represents the second norm.

[0026] Based on the same concept, the present invention also provides a target tracking method, comprising:

[0027] Calling a trained target detection model; wherein the target detection model is trained using the target detection model training method described above;

[0028] Use the trained target detection model to detect the current frame image and obtain the current frame detection result and its corresponding ReID feature map; the current frame detection result includes the current frame prediction box and its detection information;

[0029] Target tracking is performed based on historical trajectories, current frame detection results and their corresponding ReID feature maps.

[0030] Furthermore, target tracking is performed based on the historical trajectory, the current frame detection result and its corresponding ReID feature map, including:

[0031] According to the detection information of the prediction frame of the current frame, it is determined whether the current prediction frame belongs to a high-score frame or a low-score frame;

[0032] Perform a match based on the historical trajectory, high-score frame and its corresponding ReID feature map, and update and output the trajectory of the target corresponding to the high-score frame that has been successfully matched.

[0033] For historical trajectories that failed to match once, when the loss time is less than the first time threshold, a second match is performed based on the historical trajectory and the high-score frame. For the target corresponding to the high-score frame that successfully matched the second time, its trajectory is updated and output; for the target corresponding to the high-score frame that failed to match the second time, it is marked as a new trajectory and output;

[0034] Perform three matches based on the historical trajectories and low-score frames that failed to match twice. For the targets corresponding to the low-score frames that successfully matched three times, update and output their trajectories; for the low-score frames that failed to match three times, discard them.

[0035] For historical trajectories that have failed to match three times and historical trajectories that have failed to match once and whose loss time is greater than or equal to the first time threshold, when the loss time is greater than the second time threshold, the historical trajectory is discarded; when the loss time is less than or equal to the second time threshold, the historical trajectory is retained and output; wherein the second time threshold is greater than the first time threshold.

[0036] Furthermore, a match is performed based on the historical trajectory, the high-score frame and its corresponding ReID feature map, including:

[0037] Step C1: Initialize the track loss time counter t;

[0038] Step C2: determine whether the track loss time counter t is less than the second time threshold, if so, proceed to step C3; if not, output the unmatched high-resolution frame and the unmatched historical track;

[0039] Step C3: Select the trajectory with loss time t from the historical trajectory, and calculate the cosine similarity between the trajectory with loss time t and the high-score frame and its corresponding ReID feature map;

[0040] Step C4: According to the cosine similarity calculation results, the Hungarian algorithm is used to match the trajectory with a loss time of t with the target corresponding to the high-resolution frame. For the target corresponding to the high-resolution frame that is successfully matched, its trajectory is updated and output; for the high-resolution frame and historical trajectory that are not successfully matched, go to step C5;

[0041] Step C5: The track loss time counter t is increased by 1, and the process goes to step C2.

[0042] Based on the same concept, the present application also provides an electronic device, including a memory, a processor, and a computer program / instructions stored in the memory, wherein the processor executes the computer program / instructions to implement the target detection model training method or target tracking method as described above.

[0043] Based on the same concept, the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the target detection model training method or target tracking method as described above is implemented.

[0044] Based on the same concept, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the target detection model training method or target tracking method as described above.

[0045] Beneficial Effects

[0046] Compared with the prior art, the advantages of the present invention are:

[0047] The target detection model of the present invention includes not only a detection module but also a ReID module. Through a single target detection model, not only the target detection task but also the re-identification task is realized. The target detection model of the present invention effectively reduces the dependence of the re-identification task on the backbone network features by integrating the features of the backbone network at different scales, thereby achieving excellent performance of the ReID task. When performing unsupervised training on the ReID module, the present invention makes full use of the detection results of the detection module and reduces the need for manual data labeling. Compared with traditional methods, the training method of the present invention is more economical and efficient, and improves the scalability of the JDE model in different scenarios.

[0048] The tracking method of the present invention adopts different matching strategies for trajectories with different loss times by setting a first time threshold and a second time threshold, thereby effectively recovering long-term lost trajectories. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] In order to more clearly illustrate the technical solution of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only one embodiment of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0050] Figure 1 is an example diagram of using the detection result as the input of the tracker in the background technology of the present invention;

[0051] Figure 2 is an example diagram of using detection results and appearance features as input of a tracker in the background technology of the present invention;

[0052] Figure 3 is a diagram of the target detection model architecture in an embodiment of the present invention;

[0053] Figure 4 is a flow chart of a target detection model training method in an embodiment of the present invention;

[0054] Figure 5 is a flowchart of the unsupervised training of the ReID module in an embodiment of the present invention;

[0055] Figure 6 is a target tracking flow chart in an embodiment of the present invention;

[0056] Figure 7 This is a matching flow chart in an embodiment of the present invention. DETAILED DESCRIPTION

[0057] The following is a clear and complete description of the technical solutions in the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0058] The technical solution of the present application is described in detail with specific embodiments below. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0059] Embodiment 1

[0060] Figure 3 The architecture diagram of the target detection model is shown in Figure 1, where C represents concatenation, U represents upsampling, RepBlock is a neural network module composed of multiple convolutional layers, and Conv represents a convolutional layer. Figure 3 As shown in the figure, the target detection model includes a backbone network, a detection module and a ReID module. The backbone network is connected to the detection module and the ReID module. The detection module includes a first neck network and a detection head. The ReID module includes a second neck network and a ReID head. The first neck network and the second neck network use the same architecture but different parameters. The backbone network is used to extract features from the input image to obtain feature maps of different scales; the detection module is used to detect feature maps of different scales to obtain target detection results at different scales; the ReID module is used to re-identify feature maps of different scales to obtain ReID feature maps at different scales, that is, the appearance features of the target at different scales.

[0061] In this embodiment, the backbone network adopts EfficientNet; the first neck network and the second neck network both adopt the neck network in the Yolov6 model, and the ReID head adopts a single-layer convolution and is used for channel transformation. EfficientNet has outstanding structural flexibility and can adopt a multi-branch architecture, including 3×3 convolution branches, 1×1 convolution branches, and identity mapping branches. This multi-branch design enables the EfficientNet network to effectively integrate different features and receptive fields, thereby ensuring higher training accuracy than a single-branch network structure, especially in complex training scenarios and tasks. This flexibility not only improves the training effect, but also helps to maintain the high quality of the ReID feature map during the detection process. The backbone network includes three output layers, which are respectively composed of Figure 3 The three trapezoids in the figure indicate that the closer to one end of the input image, the larger the size of the output feature map. The input image is denoted as 3×W IN ×H IN, then the feature map from the first output layer is 128×(W IN / 8)×(H IN / 8), the feature map from the second output layer is 256×(W IN / 16)×(H IN / 16), and the feature map from the third output layer is 512×(W IN / 32)×(H IN / 32), where W IN Indicates the width of the input image, H IN Indicates the height of the input image.

[0062] The detection head includes a position regression head, a confidence head, and a classification head. The position regression head is used to predict the position and size of the target. Due to the size differences of the targets, the detection module detects targets of different scales by outputting feature maps of different scales. Smaller-scale feature maps contain richer semantic information and are suitable for detecting larger targets. During the training process, the loss of the position regression head is calculated by the SIoU (Scale Intersection Over Union) function to optimize the position regression accuracy. The SIoU loss includes angle loss, distance loss, shape loss, and intersection over union loss, where the angle loss is calculated as:

[0063]

[0064] Among them, Λ represents the angle loss, σ represents the distance between the real box and the predicted box, and c h Indicates the height of the minimum bounding rectangle of the real box and the predicted box, represents the center coordinate of the real box, Represents the center coordinates of the prediction box.

[0065] The distance loss is calculated as:

[0066]

[0067] γ=2-Λ (6)

[0068] Among them, Δ represents the distance loss, γ, ρ t Indicates the intermediate quantity, c w Indicates the width of the minimum bounding rectangle of the true box and the predicted box.

[0069] The shape loss is calculated as:

[0070]

[0071] Where Ω represents the shape loss, ω w ,ω h Indicates the intermediate quantity, w gt and h gtRepresent the width and height of the real box respectively, w and h represent the width and height of the predicted box respectively, θ represents the shape cost control coefficient, and its recommended value is 4.

[0072] The calculation formula of intersection-over-union loss is:

[0073]

[0074] Among them, IoU represents the intersection over union loss, B GT represents the true box, and B represents the predicted box.

[0075] The calculation formula of SIoU loss is:

[0076]

[0077] Among them, L SloU represents SIoU loss.

[0078] The confidence head is used to predict the confidence of the target, that is, the probability that the target is in the detection box, and its output value is mapped to a probability distribution of 0 to 1. During the training process, the confidence loss L is calculated by comparing with the true label using the binary cross entropy loss function. cof , which in turn guides the optimization of confidence. Targets that do not meet the confidence conditions are filtered through the confidence threshold, and the confidence loss is calculated as:

[0079]

[0080] Among them, K represents the number of samples, y k represents the true label (0 or 1) of the kth sample, p k Represents the predicted probability that the true label of the kth sample is 1.

[0081] The classification head is used to assign accurate classification labels to the objects in the prediction box. In the output layer, the classification head maps the features to the category space through a fully connected layer and outputs the probability value of each category to determine the most likely category of the object. During the training process, the classification loss is calculated by the VariFocal function, which can balance the category imbalance problem and optimize the classification accuracy. The specific calculation formula of the classification loss is:

[0082]

[0083] Among them, L cls represents the classification loss; represents the predicted probability; y represents the true label (0 or 1); α represents the adjustment factor, which is used to control the weight of positive and negative samples; ε represents the focusing parameter, which is used to adjust the influence of samples that are difficult to classify.

[0084] The target detection model of the present invention adds a ReID module on the basis of the original model. The second neck network has the same architecture as the first neck network and adopts the neck network of Yolov6, which has the characteristics of strong scale feature fusion; the detection head and the ReID head use different convolutional layers to output feature maps with different numbers of channels to complete the detection task and the ReID task. The present invention completes the detection task and the ReID task through a single target detection model. The detection module can efficiently and accurately perform target detection, provide high-quality location information, confidence assessment and classification labels, so as to achieve more accurate multi-target tracking. The second neck network of the ReID module fuses and transforms the feature maps of different scales output by the backbone network, so as to extract the appearance features of the target at different scales. These fused features are then passed to the ReID head by three different levels to obtain the target appearance representation of different scales to support more accurate target re-identification. Through this design, the ReID module can make full use of multi-scale features, thereby improving the re-identification performance of the target at different perspectives and scales, and further enhancing the stability and accuracy of target tracking.

[0085] Figure 4 The flowchart of the target detection model training method provided by the present invention is shown. Figure 4 As shown, the training method comprises the following steps:

[0086] Step A1: Freeze the ReID module and use the detection dataset to perform supervised training on the backbone network and detection module.

[0087] The detection data set includes image data and its labels. The backbone network and detection module are supervised trained using the labeled image data until the backbone network and detection module converge. During the supervised training process, the ReID module does not participate in the training, and the image data in the detection data set is discontinuous.

[0088] Step A2: Freeze the parameters of the backbone network and detection module, and use video data to perform unsupervised training on the ReID module.

[0089] Traditional ReID training is usually carried out in a supervised manner, which requires the data set to carry the ID number of the target, and this method will bring huge labor costs. The present invention adopts an unsupervised method to train the ReID module. Its core idea relies on spatial and temporal information to infer the similarity between targets, and uses strong and weak supervisory signals to guide the training of the ReID module. It can perform ReID training in unknown scenarios based on existing detection data. In the unsupervised training process of the ReID module, the backbone network and the detection module do not participate in the training, and the data set is switched to continuous video data. The video data does not need to carry any annotation information, but the images in the video data must be continuous, and the detection results of the detection module will guide the training of the ReID module. The ReID module outputs the ReID feature map corresponding to the detection results output by all detection heads, but only the ReID feature map screened out according to the detection results of the detection module is the positive sample required by the ReID module.

[0090] In a specific embodiment of the present invention, Figure 5 As shown, the video data is used to perform unsupervised training on the ReID module, including:

[0091] Step A2.1: The backbone network extracts features from continuous frame images in the video data to obtain feature maps of different scales;

[0092] Step A2.2: The detection module detects feature maps of different scales to obtain detection feature maps;

[0093] Step A2.3: The ReID module re-identifies feature maps of different scales to obtain ReID feature maps; the detection feature map and the ReID feature map only differ in the number of channels and are the same in scale, and each detection feature map corresponds to a ReID feature map;

[0094] Step A2.4: According to the detection feature map, the ReID feature map corresponding to the detection feature map is screened to obtain a ReID positive sample;

[0095] Step A2.5: Calculate the ReID loss based on the ReID positive sample set obtained from two adjacent frames of images, and adjust the parameters of the ReID module based on the ReID loss to achieve unsupervised training of the ReID module.

[0096] In step A2.4, the ReID feature graphs corresponding to the detection feature graph are not all positive samples and need to be screened. In a specific embodiment of the present invention, screening is performed by post-processing operations using the confidence of the detection feature graph and a non-maximum suppression algorithm. Specifically, determine whether the confidence of each detection feature graph is greater than a first confidence threshold. If so, retain the ReID feature graph corresponding to the detection feature graph; otherwise, remove the ReID feature graph corresponding to the detection feature graph; use a non-maximum suppression algorithm to perform overlapping removal operations on the detection feature graph corresponding to the retained ReID feature graph, and obtain a ReID feature graph corresponding to the retained detection feature graph, that is, obtain a ReID positive sample.

[0097] During the screening process, only the detection results of the detection module (i.e., the detection feature map) are used, and the ReID positive samples are screened along with the detection results.

[0098] The video data is input into the target detection model, and the ReID positive sample set obtained from the previous and next two frames is represented as E∈{e0,e1,…,e N-1 ,e N ,…,e N+M-1}, where N and M represent the number of ReID positive samples in the previous frame and the next frame respectively. The similarity matrix is ​​defined as S∈R (N+M)×(N+M+1) , the similarity matrix S in this embodiment is a cosine similarity matrix and can be expressed as The similarity matrix S has N+M rows and N+M+1 columns. The N+M rows are the sum of the number of targets in the two frames, and the N+M+1 columns are the sum of the number of targets in the two frames plus a column of placeholders. (0,N)×(0,N) and S4∈R (N,N+M)×(N,N+M) Respectively represent the cosine similarity between N targets in the previous frame image and the cosine similarity between M targets in the next frame image, and the diagonal elements are set to negative infinity because they are cosine similarities calculated for the same target; S3∈R (0,N)×(N,N+M) and S2∈R (N,N+M)×(0,N) Respectively represent the cosine similarity between the targets in the previous and next two frames; S5∈R (0,N+M)×(N+M,N+M+1) Represents a placeholder, which means there is no matching target between the two frames. The specific calculation formula for each element in the cosine similarity matrix S is:

[0099]

[0100] Among them, S ij represents the similarity between the i-th target and the j-th target, e i 、e jThey represent the i-th and j-th ReID positive samples in the ReID positive sample set E obtained from the previous and next two frames of images, respectively. represents the average value of all elements in the similarity matrix except the diagonal elements, and ||||2 represents the two-norm.

[0101] In step A2.5, the ReID loss is calculated by the internal image loss, inter-frame loss and cycle constraint loss. intra Guided by a strong supervisory signal, L intra It means that there will be no identical objects in the same image, so the cosine similarity between objects within the image should be close to 0. The specific calculation formula is:

[0102]

[0103] Inter-frame loss L inter Guided by weak supervision signals, L inter Indicates that most targets in two adjacent frames can be matched in pairs, and there will be an S in each column ij That is, if a target at the previous moment can find the target with the greatest similarity at the next moment, the cosine similarity between them should be greater than other cosine similarities, and the unmatched targets will be trained to a placeholder. The specific calculation formula for inter-frame loss is:

[0104]

[0105] Where m represents the interval value (usually the default value is 0.5), which represents the minimum distance of cosine similarity between different targets; j' represents the subscript value of the second largest element in the i-th row of the similarity matrix, j* represents the subscript value of the largest element in the i-th row of the similarity matrix, and argmax represents finding S ij J at maximum.

[0106] Cycle Constraint Loss L cycle It is used to indicate that the cosine similarity between any two objects does not change due to their position in vector multiplication. The specific calculation formula is:

[0107]

[0108] Therefore, the ReID loss is calculated as:

[0109]

[0110] The present invention utilizes the existing detection data set to train the backbone network and detection module in the target detection model, and then utilizes the video data to perform unsupervised training on the ReID module under the guidance of the detection results output by the detection module. This overcomes the limitation of the traditional method on the continuity requirement of video sequences, effectively reduces the dependence on manual labeling during the training process, and achieves the goal of training an efficient JDE model on a discrete detection data set.

[0111] Embodiment 2

[0112] The target tracking method provided by the embodiment of the present invention comprises the following steps:

[0113] Step B1: Obtain the current frame detection result and its corresponding ReID feature map.

[0114] By detecting the current frame image, the current frame detection result and its corresponding ReID feature map can be obtained, wherein the current frame detection result includes the current frame prediction box and its detection information, and the detection information includes the confidence of the current frame prediction box. The current frame image is detected using a target detection model, and the target detection model can be an existing model or a target detection model trained by the training method of Example 1 of the present application.

[0115] Step B2: Track the target based on the historical trajectory, the current frame detection result and its corresponding ReID feature map.

[0116] Target tracking is to correlate or match the historical trajectory with the current frame detection target based on the current frame detection result and its corresponding ReID feature map, so as to assign the same ID number to the targets across frames and realize multi-target tracking.

[0117] In a specific embodiment of the present invention, Figure 6 As shown in the figure, target tracking is performed based on the historical trajectory, the current frame detection result and its corresponding ReID feature map, including:

[0118] Step B2.1: Determine whether the current prediction frame belongs to a high-resolution frame or a low-resolution frame based on the detection information of the current frame prediction frame.

[0119] When the confidence of the current frame prediction frame is greater than the second confidence threshold, the current frame prediction frame is determined to be a high-resolution frame; when the confidence of the current frame prediction frame is less than or equal to the second confidence threshold, the current frame prediction frame is determined to be a low-resolution frame. All current frame prediction frames are determined to be high- and low-resolution frames, thereby obtaining a high-resolution frame set and a low-resolution frame set.

[0120] Step B2.2: Perform a match based on the historical trajectory, high-score frame and its corresponding ReID feature map. For the target corresponding to the high-score frame that has been successfully matched, update and output its trajectory; for the historical trajectory and high-score frame that have not been successfully matched, proceed to step B2.3;

[0121] Step B2.3: For a historical track that has not been successfully matched, determine whether the loss time of the historical track is less than the first time threshold T short If yes, go to step B2.4; otherwise, go to step B2.6;

[0122] Step B2.4: Perform secondary matching based on historical trajectories and high-score frames. For the targets corresponding to the high-score frames that have successfully matched the secondary, update and output their trajectories; for the targets corresponding to the high-score frames that have not successfully matched the secondary, mark them as new trajectories and output them; for the historical trajectories that have not successfully matched the secondary, proceed to step B2.5;

[0123] Step B2.5: Perform three matches based on the historical trajectories and low-score frames that failed to match twice. For the targets corresponding to the low-score frames that successfully matched three times, update and output their trajectories; discard the low-score frames that failed to match three times; and proceed to step B2.6 for the historical trajectories that failed to match three times;

[0124] Step B2.6: For a historical track that has failed to match three times and a historical track that has failed to match once and whose loss time is greater than or equal to the first time threshold, determine whether the loss time of the historical track is greater than the second time threshold T long If yes, go to step B2.7; otherwise, go to step B2.8;

[0125] Step B2.7: discard the historical track;

[0126] Step B2.8: Keep the historical track and output it.

[0127] The matching stage to be participated in by the historical trajectory is determined by the first time threshold and the second time threshold. long Greater than the first time threshold T short , the first time threshold T short and the second time threshold T long In units of image frames, the first time threshold of this embodiment is set to 30 frames, and the second time threshold can be dynamically adjusted according to the tracking scene. If the target in the scene is lost for too long, the second time threshold is set to a larger value. The second time threshold of this embodiment is set to 200 frames.

[0128] Execute steps B2.1 to B2.8 to complete a round of matching. Each round of matching includes three matches, namely, high-score ReID cascade matching (corresponding to step B2.2), high-score IoU matching stage (corresponding to step B2.4) and low-score matching stage (corresponding to step B2.5). All output trajectories are used as historical trajectories for the next round of matching. The task of a single round of matching is to assign the target detected in the current frame to the historical trajectory and update the historical trajectory; the historical trajectory is the target trajectory that has been formed before the current frame and its corresponding ReID feature map. From the time a trajectory fails to successfully match any detected target in a single round of matching, its loss time is recorded. If there is still no subsequent match, the loss time is increased by 1 after each round of matching; if the re-match is successful, the loss time is reset to 0.

[0129] In a specific embodiment of the present invention, Figure 7 As shown in the figure, a match is performed based on the historical trajectory, the high-score frame and its corresponding ReID feature map, including:

[0130] Step C1: Initialize the track loss time counter t, the initial value of the track loss time counter i is 0;

[0131] Step C2: Determine whether the track loss time counter t is less than the second time threshold. If so, proceed to step C3; if not, output the unmatched high-resolution frame and the unmatched historical track, and proceed to step B2.3;

[0132] Step C3: Select the trajectory with loss time t from the historical trajectory, and calculate the cosine similarity between the trajectory with loss time t and the high-score frame and its corresponding ReID feature map;

[0133] Step C4: According to the cosine similarity calculation results, the Hungarian algorithm is used to match the trajectory with a loss time of t with the target corresponding to the high-resolution frame, that is, for the target corresponding to the high-resolution frame that is successfully matched, its trajectory is updated and output; for the high-resolution frame and historical trajectory that are not successfully matched, go to step C5;

[0134] Step C5: Add 1 to the track loss time counter t and go to step C2.

[0135] The present invention can be applied to vehicle target detection and tracking. When the target detection model is trained, the images in the detection data set are images containing vehicles, and the video data are vehicle motion videos.

[0136] The present invention adopts different matching strategies for trajectories with different loss times by setting a first time threshold and a second time threshold, effectively recovers long-term lost trajectories, improves the stability and accuracy of target tracking, reduces the adverse effects caused by Kalman filter prediction errors, and thus improves the accuracy of target re-identification and matching.

[0137] Embodiment 3

[0138] An embodiment of the present application also provides an electronic device, which includes: a memory, a processor, and a computer program / instructions stored in the memory, and the processor executes the computer program / instructions to implement the target detection model training method or target tracking method in the embodiment of the present application.

[0139] Although not shown, the electronic device includes a processor, which can perform various appropriate operations and processes according to the program and / or data stored in the read-only memory (ROM) or the program and / or data loaded from the storage portion into the random access memory (RAM). The processor can be a multi-core processor, or it can include multiple processors. In some embodiments, the processor can include a general main processor and one or more special coprocessors, such as a central processing unit, a graphics processing unit (GPU), a neural network processor (NPU), a digital signal processor (DSP), etc. In RAM, various programs and data required for device operation are also stored. The processor, ROM and RAM are connected to each other via a bus. The input / output (I / O) interface is also connected to the bus.

[0140] The processor and memory are used together to execute the program / instructions stored in the memory. When the program / instructions are executed by the computer, the methods, steps or functions described in the above embodiments can be implemented.

[0141] Although not shown, an embodiment of the present application also provides a computer-readable storage medium on which a computer program / instruction is stored. When the computer program / instruction is executed by a processor, the target detection model training method or the target tracking method in the embodiment of the present application is implemented.

[0142] Readable storage media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer-readable media does not include temporary computer-readable media (transitory media), such as modulated data signals and carrier waves.

[0143] Although not shown, an embodiment of the present application also provides a computer program product, including: a computer program / instruction, which, when executed by a processor, implements the target detection model training method or target tracking method in the embodiment of the present application.

[0144] What is disclosed above is only a specific implementation mode of the present invention, but the protection scope of the present invention is not limited thereto. Any technician familiar with the technical field can easily think of changes or modifications within the technical scope disclosed in the present invention, which should be covered within the protection scope of the present invention.

Claims

1. A target detection model training method, characterized in that: The target detection model includes a backbone network, a detection module and a ReID module, the backbone network is connected to the detection module and the ReID module, the detection module includes a first neck network and a detection head, the ReID module includes a second neck network and a ReID head, and the second neck network has the same architecture as the first neck network; the training method includes: Freeze the ReID module, and use the detection data set to perform supervised training on the backbone network and the detection module; wherein the detection data set includes image data and its labels; The parameters of the backbone network and the detection module are frozen, and the ReID module is trained unsupervisedly using video data.

2. The target detection model training method according to claim 1, characterized in that: The ReID module is unsupervisedly trained using video data, including: The backbone network extracts features from continuous frame images in the video data to obtain feature maps of different scales; The detection module detects feature maps of different scales to obtain detection feature maps; The ReID module re-identifies feature maps of different scales to obtain ReID feature maps; wherein each detection feature map corresponds to a ReID feature map; According to the detection feature graph, a ReID feature graph corresponding to the detection feature graph is screened to obtain a ReID positive sample; The ReID loss is calculated according to the ReID positive sample set obtained from two adjacent frames of images, and the parameters of the ReID module are adjusted according to the ReID loss to achieve unsupervised training of the ReID module.

3. The target detection model training method according to claim 2, characterized in that: Screening a ReID feature map corresponding to the detection feature map according to the detection feature map includes: Determine whether the confidence of the detection feature map is greater than a first confidence threshold, if so, retain the ReID feature map corresponding to the detection feature map; otherwise, remove the ReID feature map corresponding to the detection feature map; An overlapping elimination operation is performed on the detection feature map corresponding to the retained ReID feature map to obtain the ReID feature map corresponding to the retained detection feature map, that is, the ReID positive sample is obtained.

4. The target detection model training method according to claim 2 or 3, characterized in that: The specific calculation formula of the ReID loss is: Among them, L ID represents the ReID loss, L intra Represents internal loss, L inter Indicates the inter-frame loss, L cycle represents the cycle constraint loss; N represents the number of positive ReID samples obtained from the previous frame of two adjacent frames, and M represents the number of positive ReID samples obtained from the next frame of two adjacent frames; S ij Represents the similarity between the i-th target and the j-th target, which is composed of all similarities S ij and a column of placeholders to form a similarity matrix S. The similarity matrix S has N+M rows and N+M+1 columns. m represents the interval value, j' represents the subscript value of the second largest element in the i-th row of the similarity matrix S, j* represents the subscript value of the largest element in the i-th row of the similarity matrix S, and argmax represents finding the value of S. ij j at maximum; e i 、e j They represent the i-th and j-th ReID positive samples in the ReID positive sample set obtained from two adjacent frames of images respectively; represents the average value of all elements in the similarity matrix S except the diagonal elements, and || ||2 represents the two-norm.

5. A target tracking method, characterized in that: The tracking method includes: Calling a trained target detection model; wherein the target detection model is trained using the target detection model training method according to any one of claims 1 to 4; Use the trained target detection model to detect the current frame image and obtain the current frame detection result and its corresponding ReID feature map; the current frame detection result includes the current frame prediction box and its detection information; Target tracking is performed based on historical trajectories, current frame detection results and their corresponding ReID feature maps.

6. The target tracking method according to claim 5, characterized in that: Target tracking is performed based on historical trajectories, current frame detection results and their corresponding ReID feature maps, including: According to the detection information of the prediction frame of the current frame, it is determined whether the current prediction frame belongs to a high-score frame or a low-score frame; Perform a match based on the historical trajectory, high-score frame and its corresponding ReID feature map, and update and output the trajectory of the target corresponding to the high-score frame that has been successfully matched. For historical trajectories that failed to match once, when the loss time is less than the first time threshold, a second match is performed based on the historical trajectory and the high-score frame. For the target corresponding to the high-score frame that successfully matched the second time, its trajectory is updated and output; for the target corresponding to the high-score frame that failed to match the second time, it is marked as a new trajectory and output; Perform three matches based on the historical trajectories and low-score frames that failed to match twice. For the targets corresponding to the low-score frames that successfully matched three times, update and output their trajectories; for the low-score frames that failed to match three times, discard them. For historical trajectories that have failed to match three times and historical trajectories that have failed to match once and whose loss time is greater than or equal to the first time threshold, when the loss time is greater than the second time threshold, the historical trajectory is discarded; when the loss time is less than or equal to the second time threshold, the historical trajectory is retained and output; wherein the second time threshold is greater than the first time threshold.

7. The target tracking method according to claim 6, characterized in that: A match is performed based on the historical trajectory, high-score frame and its corresponding ReID feature map, including: Step C1: Initialize the track loss time counter t; Step C2: determine whether the track loss time counter t is less than the second time threshold, if so, proceed to step C3; if not, output the unmatched high-resolution frame and the unmatched historical track; Step C3: Select the trajectory with loss time t from the historical trajectory, and calculate the cosine similarity between the trajectory with loss time t and the high-score frame and its corresponding ReID feature map; Step C4: According to the cosine similarity calculation results, the Hungarian algorithm is used to match the trajectory with a loss time of t with the target corresponding to the high-resolution frame. For the target corresponding to the high-resolution frame that is successfully matched, its trajectory is updated and output; for the high-resolution frame and historical trajectory that are not successfully matched, go to step C5; Step C5: The track loss time counter t is increased by 1, and the process goes to step C2.

8. An electronic device comprising a memory, a processor, and a computer program / instruction stored in the memory, characterized in that: The processor executes the computer program / instructions to implement the target detection model training method according to any one of claims 1 to 4 or the target tracking method according to any one of claims 5 to 7.

9. A computer-readable storage medium having a computer program / instruction stored thereon, characterized in that: When the computer program / instruction is executed by a processor, the target detection model training method as described in any one of claims 1 to 4 or the target tracking method as described in any one of claims 5 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instruction is executed by a processor, the target detection model training method as described in any one of claims 1 to 4 or the target tracking method as described in any one of claims 5 to 7 is implemented.

Images