Method and system for constructing multi-face tracking dataset based on Retinaface and DeepSort algorithms

Abstract

The present invention relates to the technical field of multi-face tracking, and more particularly to a method and system for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms. The method comprises constructing a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm, automatically annotating multi-face tracking data, automatically cleaning the multi-face tracking annotated data using a face dataset cleaning method based on the DBSCAN clustering algorithm, manually cleaning the multi-face tracking annotated data based on the automatic cleaning of the multi-face tracking annotated data, and performing logic and format checks on the cleaned multi-face tracking annotated data to eliminate logic errors and situations that do not conform to the face tracking annotated format. The present invention can efficiently generate a high-quality face tracking dataset.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-face tracking, and in particular to a method and system for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms. Background Art

[0002] The multi-face tracking dataset is a specific research direction for multi-target tracking datasets. It can be used to train face tracking models based on deep learning technology and provide a fair testing platform for various face tracking algorithms to compare their performance. Currently, the lack of datasets specifically for face tracking has limited the research and application of face tracking technology.

[0003] Current research on face tracking datasets faces two major challenges. First, there are too few datasets dedicated to face tracking. Most current tracking datasets are used for single-target or multi-target tracking. Face tracking is only a specific research area within the target tracking field. While target tracking datasets for pedestrians, vehicles, animals, and other objects are already plentiful, datasets and performance evaluation metrics for face tracking tasks are relatively scarce. Even the Chokepoint face tracking dataset, which is widely used in the industry, only annotates the left and right eye positions of pedestrians in simple scenes (with only one pedestrian in a single frame). However, it does not provide annotated data for the six video sequences ("P2E_S5" and "P2L_S5"), which include complex scenes with multiple pedestrians appearing simultaneously and occluding each other. This face tracking dataset needs further improvement. Second, face tracking datasets are expensive to produce, have inconsistent standards, and lack a systematic approach to constructing face tracking datasets. Currently, most tracking datasets are manually created using annotation tools. Common tools include LabelImg, DarkLabel, and X-Anylabeling. Manual annotation is costly, significantly limiting the scale of dataset creation. For example, Barquero G et al., while conducting research on multi-face tracking technology, lacked suitable multi-face tracking data. They combined face detection with manual annotation to annotate the "P2E_S5" and "P2L_S5" data sets from Chokepoint and selected YouTube videos. However, manually associating a large number of face tracks would undoubtedly be prohibitively expensive. The VggFace2 dataset, a well-known dataset in the field of face recognition, took nearly five months to construct, with nearly four months dedicated to manual annotation and verification. The CPLFW dataset, used for cross-age face recognition research, required the participation of over 300 volunteers for its annotation and verification. Furthermore, manually annotated datasets vary in quality and standard, inevitably containing mislabeled and missing data. For example, research by Fei.W et al. shows that the CASIA-Webface, MS-Celeb-1M, and MegaFace datasets used for face recognition task research contain a large number of labeling errors and duplicate data. Summary of the Invention

[0004] The present invention aims to solve the current problems of high manual annotation cost, inconsistent annotation quality and low efficiency of face tracking datasets. A method and system for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms are proposed, which can generate high-quality multi-face tracking datasets efficiently and at low cost.

[0005] In order to achieve the above purpose, the technical solutions adopted are:

[0006] A method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, including:

[0007] Build a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm to automatically annotate multi-face tracking data;

[0008] The face dataset cleaning method based on DBSCAN clustering algorithm is used to automatically clean the multi-face tracking and annotation data;

[0009] On the basis of automatic cleaning of multi-face tracking and annotation data, manual cleaning of multi-face tracking and annotation data is mainly carried out by manual proofreading;

[0010] Perform logic and format checks on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

[0011] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, the working process of the multi-face tracking framework is as follows:

[0012] First, the DeepSort multi-target tracking algorithm is used to determine the facial motion trajectory of each object. Then, the determined facial motion trajectory ID of each object is associated with the facial bounding box detected by the Retinaface face detection algorithm to automatically generate the true value box information of the face tracking data.

[0013] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, the Retinaface face detection model includes a feature pyramid network, a context head module and a cascaded multi-task loss function. First, the feature pyramid network obtains the face image and outputs feature maps of different scales. Then, the context head module predicts the face bounding box. Finally, the cascaded multi-task loss function is used to train the model. The DeepSort multi-target tracking algorithm first uses the Retinaface algorithm to generate a face bounding box for each frame of the video. Then, the appearance features of each face target are extracted through a deep learning model. Then, the data association method is used to match the detection result of the current frame with the tracked face target of the previous frame. Finally, the tracking result of each face target in the video sequence is output.

[0014] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, the face bounding box information format is:<bb_left> ,<bb_top> ,<bb_right> ,<bb_bottom> , <conf> , <cons>, where represents the frame number,<bb_left> Indicates the horizontal coordinate of the upper left corner of the bounding box,<bb_top> Indicates the ordinate of the upper left corner of the bounding box,<bb_right> Indicates the horizontal coordinate of the lower right corner of the bounding box;<bb_bottom> Indicates the ordinate of the lower right corner of the bounding box, <conf>represents the confidence of the bounding box, <cons>Indicates whether to consider the detection information; the format of the face tracking data truth box information is:, <id>,<bb_left>,<bb_top>,<bb_right>,<bb_bottom>, <cons>,in <id>Indicates the track ID. The face track ID number of the same object is the same; <cons>Indicates whether to consider the true value information; the rest of the meaning is the same as explained in the above face bounding box information.

[0015] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, a face dataset cleaning method based on the DBSCAN clustering algorithm is used to automatically clean the multi-face tracking annotation data, specifically including:

[0016] Extract all face images in the video sequence based on the automatic annotation data generated by the multi-face tracking framework, and save the face images with the same face track ID in the same folder;

[0017] Rename the image, and associate the new image name with the face category and face track ID;

[0018] Resize all face images to the same size and perform histogram equalization on the images;

[0019] The clarity of the image is evaluated by calculating the Laplace variable of the face image, and the clearest face image is used as the representative face image in the face trajectory;

[0020] Use the DBSCAN clustering algorithm to cluster the representative face images in each trajectory;

[0021] According to the image name, the face images that did not participate in the clustering operation are reallocated to their respective categories;

[0022] According to the cleaned face categories and the name indexes of the face images in each category, the initial multi-face tracking and annotation data is automatically corrected to obtain the cleaned multi-face tracking and annotation data.

[0023] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, manually cleaning the multi-face tracking annotation data specifically includes:

[0024] First, we manually compare videos with facial bounding boxes and trajectory IDs, record the associated indexes of the facial trajectory IDs that need to be corrected, and use script tools to automatically correct the annotated data based on the associated indexes, thereby correcting the problem of multiple IDs corresponding to the same face.

[0025] Then, by manually comparing videos with face bounding boxes and trajectory IDs, we used Darklabel annotation software and script tools to correct problems such as missing faces and different faces corresponding to the same ID.

[0026] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, in order to unify the annotation format of the multi-face tracking dataset, the following manual annotation rules are determined: ① standardize the annotation range of the face bounding box; ② define all faces that can be detected by the Retinaface algorithm or have recognizable facial features as valid faces, and select valid faces for annotation; ③ standardize the starting and ending positions of the face track; ④ face occlusion defines the face occlusion situation that causes the face tracking track ID to switch.

[0027] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, the judgment condition for face occlusion is further defined as follows: the face of an object has been successfully detected, until a certain frame, the face of the object suddenly cannot be detected, and the face bounding box of the object in the previous frame is still within the monitoring field of view; another case is that the coordinates of the face bounding box exceed the monitoring field of view.

[0028] Furthermore, the present invention also provides a multi-face tracking dataset construction system based on Retinaface and DeepSort algorithms, which includes an automatic labeling module, an automatic cleaning module, a manual cleaning module and a logical format verification module, wherein:

[0029] Automatic labeling module, used to build a multi-face tracking framework based on Retinaface face detection algorithm and DeepSort multi-target tracking algorithm, and automatically label multi-face tracking data;

[0030] Automatic cleaning module, used to automatically clean multi-face tracking and annotation data using a face dataset cleaning method based on the DBSCAN clustering algorithm;

[0031] The manual cleaning module is used to manually clean the multi-face tracking and annotation data based on automatic cleaning, mainly through manual proofreading;

[0032] The logic format verification module is used to perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

[0033] The beneficial effects achieved by adopting the above technical solution are:

[0034] The present invention utilizes a face detection algorithm and a multi-target tracking algorithm to design a multi-face tracking framework, and utilizes the framework to automatically and efficiently preliminarily generate multi-face tracking annotation data. The present invention innovatively proposes a face data set cleaning method based on the DBSCAN clustering algorithm, which greatly reduces the amount of data involved in face clustering, improves clustering efficiency while ensuring the clustering effect, and effectively identifies and merges erroneous data generated by factors such as temporary face occlusion or resolution changes during the tracking process, thereby generating high-quality multi-face tracking annotation data. The present invention further improves the quality of the data set by manually cleaning the multi-face tracking annotation data automatically generated and cleaned by the multi-face tracking framework, mainly manually correcting faces that are not detected and tracked by the multi-face tracking framework and face trajectories that are tracked incorrectly. Finally, the cleaned annotation data is formatted and logically checked to eliminate logical errors and detect situations that do not conform to the multi-face tracking data annotation format. The present invention not only greatly reduces the workload of manual labeling of multi-face tracking data, improves the quality and efficiency of labeled data, but also further enriches the face data production method. The method can also be used to produce face recognition data and face re-identification data with a large number of pseudo-labels from video sequences, promoting the research of face recognition and face re-identification algorithms in applications such as intelligent monitoring; the method can also further expand the scope of application. By simply replacing the target detection module, it can be used to construct various multi-target tracking data sets, providing a new idea and reference for the research on target tracking data set construction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention. The drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0036] Figure 1 1 is a flow chart of a method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to an embodiment of the present invention;

[0037] Figure 2 Schematic diagram of the feature pyramid network structure according to an embodiment of the present invention;

[0038] Figure 3 1 is a schematic diagram of the composition of the cascaded multi-task loss function according to an embodiment of the present invention;

[0039] Figure 4 1 is a flow chart of the DeepSort multi-target tracking algorithm according to an embodiment of the present invention;

[0040] Figure 5 This is a schematic diagram of the automatic labeling process of multi-face tracking data according to an embodiment of the present invention;

[0041] Figure 6 This is a schematic diagram of the automatic cleaning process of multi-face tracking and annotation data according to an embodiment of the present invention;

[0042] Figure 7 Schematic diagram of density description of the DBSCAN clustering algorithm according to an embodiment of the present invention;

[0043] Figure 8 This is a schematic diagram of automatically cleaning multi-face tracking and annotation data using a clustering algorithm according to an embodiment of the present invention;

[0044] Figure 9 This is a schematic diagram of the manual cleaning process of multi-face tracking and annotation data according to an embodiment of the present invention;

[0045] Figure 10 This is an example diagram of different faces labeled with the same ID according to an embodiment of the present invention;

[0046] Figure 11 This is an example diagram of repairing an incorrectly labeled ID during manual cleaning according to an embodiment of the present invention;

[0047] Figure 12 This is an example diagram of the same face being labeled with different IDs according to an embodiment of the present invention;

[0048] Figure 13 This is an example diagram of repairing an incorrectly labeled ID during manual cleaning according to an embodiment of the present invention;

[0049] Figure 14 Schematic diagram of four types of face trajectories according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the accompanying drawings of specific embodiments of the present invention to clearly and completely describe the exemplary embodiments of the present invention. Unless otherwise defined, technical or scientific terms used in the present invention should be given the common meanings understood by people with ordinary skills in the relevant field.

[0051] The multi-face tracking dataset proposed in this embodiment is produced by combining automatic labeling with manual labeling, which improves the efficiency of dataset production while taking into account the quality of the dataset. The basic idea of ​​automatic labeling is to automatically generate face tracking labeling data based on the face tracking algorithm. Specifically, the face detection algorithm is first used to generate face bounding box information to replace the true value box information that originally needed to be manually labeled, and then the face motion trajectory ID determined by the face tracking algorithm is used to replace the face motion trajectory ID information that originally needed to be manually labeled. The produced multi-face tracking dataset includes two parts: a training set train and a test set test. The training set contains the face bounding box information det_train and the true value box information gt_train of the objects in each video, and the test set only contains the face bounding box information det_test of the objects in the test video. This embodiment discloses a method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, such as Figure 1 As shown, the following steps are included:

[0052] Step S101: construct a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm to automatically annotate multi-face tracking data.

[0053] The multi-face tracking framework first uses the DeepSort multi-target tracking algorithm to determine the facial motion trajectory of each object. It then associates the determined facial motion trajectory ID with the face bounding boxes detected by the Retinaface face detection algorithm, thereby automatically generating the ground truth frame gt_train information for the face tracking data. The det_traint and det_test face bounding box information are automatically generated by the Retinaface face detection algorithm.

[0054] The Retinaface face detection model includes a feature pyramid network, a context head module, and a cascaded multi-task loss function. First, the feature pyramid network obtains the input face image and outputs 5 feature maps of different scales; then, the context head module predicts the face bounding box. Among them, the first context head module predicts the face bounding box through regular anchors, and the second context head module predicts a more accurate face bounding box through regression anchors. The second context head module is generated by the first context head module. Finally, according to the context head module, Retinaface combines the loss functions of face classification, face box regression, 5 face true value coordinate regression, and 1000 3D vertex regression into a cascaded multi-task loss function for model training. The feature pyramid network structure is as follows: Figure 2 As shown, the cascade multi-task loss function structure is as follows Figure 3 shown.

[0055] The DeepSort algorithm is a classic multi-target tracking algorithm, which is improved from the Sort algorithm. Since the Sort algorithm only considers the target motion information, it is easy to lose tracking when the target is occluded. Therefore, the DeepSort algorithm adds appearance information on the basis of motion information, that is, adds the appearance feature information of the face target extracted by the neural network to improve the tracking accuracy when the face is occluded. The operation process of the DeepSort multi-target tracking algorithm includes target detection, feature extraction, data association and state update. First, the Retinaface algorithm is used to identify the face target in each frame of the video and generate a face bounding box; then the deep learning model extracts the appearance features of each face target; then, the data association method (such as the Hungarian algorithm) is used to match the detection result of the current frame with the tracked face target of the previous frame, considering the motion model and appearance features of the face target; finally, the state of the face target is updated, the trajectory is managed, and the tracking result of each face target in the video sequence is output. The process of the DeepSort multi-target tracking algorithm is as follows Figure 4 shown.

[0056] In order to accurately describe the frame number, object ID, location and other information of the face bounding box in each video frame, a label data is set for the face image in each frame of the video. Using the above multi-face tracking technology, each face track is automatically labeled. The automatic labeling process of multi-face tracking data is as follows: Figure 5 There are two main types of annotation formats in face tracking datasets: one is face detection annotation information, and the other is the true value information of face trajectory.

[0057] The face bounding box information format is:

[0058] ,<bb_left> ,<bb_top> ,<bb_right> ,<bb_bottom> , <conf> , <cons>, where represents the frame number,<bb_left> Indicates the horizontal coordinate of the upper left corner of the bounding box,<bb_top> Indicates the ordinate of the upper left corner of the bounding box,<bb_right> Indicates the horizontal coordinate of the lower right corner of the bounding box;<bb_bottom> Indicates the ordinate of the lower right corner of the bounding box, <conf>represents the confidence of the bounding box, <cons>Indicates whether to consider this detection information.

[0059] The format of the true value frame information of face tracking data is:

[0060] , <id>,<bb_left>,<bb_top>,<bb_right>,<bb_bottom>, <cons>,in <id>Indicates the track ID. The face track ID number of the same object is the same; <cons>Indicates whether to consider the true value information; the rest of the meaning is the same as explained in the above face bounding box information.

[0061] The data cleaning of this embodiment is generally divided into two steps. First, "automatic cleaning of multi-face tracking and annotation data" is performed. Its core is to use the face clustering algorithm to clean the facial images in various categories generated based on the face tracking data; then "manual cleaning of multi-face tracking and annotation data" is performed, which mainly cleans the multi-face tracking data annotation files after automatic cleaning, including cleaning of missed labels, multiple labels, and mislabeled data that were not successfully corrected in the automatic cleaning stage, to further improve the quality of the dataset.

[0062] Step S102: Automatically clean the multi-face tracking and annotation data using a face data set cleaning method based on the DBSCAN clustering algorithm.

[0063] "Automatic face data cleaning" is performed after preprocessing the face images by unifying the image size and filtering the blurred images. It mainly uses the DBSCAN clustering algorithm to cluster the representative faces in each track. This method conforms to the law of generating face track images by the multi-face tracking framework, that is, the face images under the same track are likely to belong to the same person. Even if there are multiple faces of the same track, they can be corrected in the manual cleaning stage. Therefore, during automatic cleaning, only face images of different tracks need to be clustered, so as to reduce the number of face images participating in the clustering and improve the computational efficiency and clustering quality. According to the face categories after face clustering cleaning and the face image name index in each category, the labeled data generated by the multi-face tracking framework is automatically corrected. The automatic cleaning process of multi-face tracking labeled data is as follows: Figure 6 The specific operation process is as follows:

[0064] (1) Extract all face images in the video sequence based on the automatic annotation data generated by the multi-face tracking framework, and save the face images with the same face track ID in the same folder.

[0065] (2) Rename the image and associate the new image name with the face category and face trajectory ID to facilitate subsequent classification.

[0066] (3) All face images are resized to the same size to eliminate the impact of size differences on clustering results.

[0067] (4) Perform histogram equalization on the image to enhance the contrast of the image and reduce the brightness difference between different images.

[0068] (5) The clarity of the image is evaluated by calculating the Laplace variable of the face image. The unclear face image is deleted through a custom threshold, and the clearest face image is copied. This face image is called the representative face data in the face track.

[0069] (6) The DBSCAN clustering algorithm is used to cluster the representative face images in each trajectory, and further correctly cluster the face images of the wrong category generated by the multi-face tracking framework.

[0070] (7) According to the image names, the face images that did not participate in the clustering operation are reallocated to their respective categories.

[0071] (8) Based on the cleaned face categories and the name index of the face images in each category, the initial multi-face tracking and annotation data is automatically corrected using the script to obtain the cleaned multi-face tracking and annotation data.

[0072] During the automatic facial data cleaning process, the data generated by the multi-target tracking algorithm is continuous in time and space. Considering that facial images in the same facial track are likely to belong to the same person, and facial images in different facial tracks within the same frame definitely do not belong to the same person, only one representative facial data piece is selected from each facial track of the automatically generated data for face clustering during face clustering. This greatly reduces the computational complexity of the clustering algorithm and also mitigates the impact of the increased density of a large amount of facial data in the feature space, which can lead to poor clustering results. In this case, the DBSCAN clustering algorithm was selected to cluster the facial data for data cleaning purposes. A significant advantage of the DBSCAN clustering algorithm is that it does not require the user to predefine the number of clusters. This flexibility allows it to perform well when the number of clusters is unknown. Furthermore, DBSCAN has the ability to identify and process noise and outliers, a feature that is particularly critical in facial image clustering, as image data often contains multiple interfering elements. Although the parameter setting of DBSCAN has some influence on the final clustering effect, its design allows users to adjust the parameters according to the specific data set and clustering target, thus providing greater adaptability. Finally, DBSCAN can also effectively handle high-dimensional data, which is an indispensable advantage for the face image clustering task containing a large amount of feature information. Figure 7 As shown in the figure, the clustering algorithm automatically cleans the multi-face tracking and annotation data. Figure 8 shown.

[0073] Step S103: Based on the automatic cleaning of the multi-face tracking and annotation data, the multi-face tracking and annotation data is manually cleaned mainly by manual proofreading and supplemented by script tools. This mainly deals with the face track ID mislabeling, missing labeling, and multiple labeling that were not successfully corrected in the automatic cleaning stage. The manual cleaning process of the multi-face tracking and annotation data is as follows: Figure 9 The manual cleaning process is as follows:

[0074] (1) By manually comparing videos with face bounding boxes and trajectory IDs, the associated index of the face trajectory ID that needs to be corrected is recorded, and the script tool is used to automatically correct the labeled data based on the associated index to correct the problem of multiple IDs corresponding to the same face.

[0075] (2) By manually comparing videos with face bounding boxes and trajectory IDs, the Darklabel annotation software and script tools are used to correct the problems of missing faces and different faces corresponding to the same ID.

[0076] Through further manual cleaning and testing, more accurate multi-face tracking and annotation data is obtained. The following details the process of manually handling face data mislabeling, face track ID omissions, and face track ID duplication.

[0077] ① Facial data mislabeling. This case divides facial data mislabeling into two categories: facial track ID mislabeling and facial detection frame mislabeling. Facial track ID mislabeling can be divided into two situations. The first situation is that the same ID is assigned to different people one after another; the other situation is that different IDs are assigned to the same person one after another. The first situation of facial track ID mislabeling is as follows: Figure 10 As shown in the figure, ID 28 is assigned to two different face targets in succession. During the manual labeling process, if the target with the larger number of frames to which the face ID is incorrectly assigned is the first labeled ID, the incorrect ID will be corrected to the next digit of the last ID number in the current labeled data. For example, if the last ID in the current labeled data is 96, then the "manual labeling auxiliary script" can be used to correct all incorrect IDs to 97. Figure 11 As shown, the new labeled data is obtained. The next time you manually label, the last ID is found based on the latest labeled data and the wrong ID is modified, and so on. The second case of face track ID mislabeling is as follows Figure 12 As shown in the figure, when the same face target is marked with different IDs, the corresponding correct and incorrect IDs in the frames are manually recorded, and then the "face tracking and annotation data correction script" is used to automatically correct them in batches. The corrected face track ID is as follows Figure 13 Face detection frame mislabeling means that due to various factors, the face detection algorithm cannot correctly detect the face, resulting in obvious deviations in the face bounding box. In this case, you can use Darklabel annotation software to assist in annotation and generate face bounding box data. Depending on the amount of correction work required, you can choose to add the corrected data to the annotation data file manually or with the help of a script.

[0078] ② Missing face track IDs. The missing face tracks referred to in this case fall into two categories: one is due to the face detection algorithm failing to correctly detect faces in the video, resulting in a failure to track those faces; the other is due to the filtering out of low-quality face data during the automatic cleaning phase to improve clustering quality and efficiency. For the first case, Darklabel software can be used to label the data according to manual labeling rules. For the second case, refer to the face track ID mislabeling case in ① and use the "Face Tracking Labeling Data Correction Script" and "Manual Labeling Assistance Script" to correct it.

[0079] ③ Over-labeling of face track IDs. In this case, over-labeling of face track IDs refers to the labeling of unnecessary face tracks when a face image appears on a reflective surface such as glass or water. In this case, the redundant face track IDs can be deleted manually or with the help of an auxiliary script.

[0080] In order to further standardize manual cleaning standards, the following manual labeling rules are determined.

[0081] ① Determination of the face bounding box. To unify the annotation rules, this case standardizes the annotation range of the face bounding box, requiring the face bounding box to be aligned with the range of the face as accurately as possible. The bounding box should include all pixels of the left and right eyes, nose, and left and right corners of the mouth. For a face in a frontal view with no obstructions, the tip of the nose is approximately located at the center of mass of the face bounding box, the left and right side bounding boxes are between the corners of the eyes and the ears, the upper bounding box and the horizontal line passing through the tip of the nose are roughly symmetrical about the horizontal line between the eyebrows, the lower bounding box is flush with the lower jaw of the face, and the aspect ratio of the face bounding box is approximately 3:4; for a face in a side view or that is obscured, at least one eye, the tip of the nose, and one corner of the mouth should be ensured. Otherwise, it is considered an invalid face image and the face bounding box will not be annotated. The upper and lower bounding boxes are consistent with the rules in the frontal view.

[0082] ② Valid face conditions. All faces that can be detected by the Retinaface algorithm or whose facial features can be identified are called valid faces. To ensure that various face tracking algorithms can fully utilize their performance when evaluated using this dataset, all valid faces are annotated. When evaluating the performance of face tracking algorithms, each algorithm can choose an appropriate face size for evaluation based on its own characteristics. As long as all algorithms are evaluated under the same conditions, the final results are comparable.

[0083] ③ The start and end of the trajectory. The starting position of the face trajectory is usually determined by the face detection algorithm. When the face of an object is detected for the first time, the corresponding frame is considered to be the starting position of the face trajectory of the object. If there is a small face that is not detected by the face detection algorithm and meets the above-mentioned "valid face conditions" standard, the face in the frame is manually marked as the starting position of the trajectory. In order to be applicable to the evaluation of face tracking algorithms with face re-identification functions, when the face is occluded or the face frame coordinates exceed the monitoring field of view, the trajectory ID is always kept unchanged to prevent the face corresponding to the ID from reappearing until the last frame of the video sequence. Therefore, the end position of a trajectory is the frame corresponding to the last successful detection in the trajectory.

[0084] ④ Face occlusion. "Face occlusion" here defines the face occlusion that causes the face tracking trajectory ID to switch, and does not include occlusions that have never been detected by the face detection algorithm. Judgment conditions for face occlusion: the face of an object has been successfully detected, until a certain frame, the face of the object suddenly cannot be detected, and the face frame of the object in the previous frame is still within the entire monitoring view range; another special case of occlusion is that the face frame coordinates exceed the monitoring view range. The judgment conditions are: the coordinates of any vertex of the bounding box are negative or the horizontal coordinate is greater than the maximum horizontal coordinate of the monitoring view, or the vertical coordinate is greater than the maximum vertical coordinate of the monitoring view. When face occlusion occurs, it is easy to cause face ID switching or trajectory interruption, resulting in track fragmentation, such as Figure 14 shown.

[0085] Step S104: Verify the logic and format of the multi-face tracking annotation data. Because human error can occur during the manual cleaning phase, resulting in a small amount of incorrectly formatted annotation data, or logical errors such as "multiple identical face tracking frames in the same video frame" or "multiple identical track IDs in the same video frame," the "annotation data verification script" provided in this application is used to perform format and logic verification on the cleaned annotation data to eliminate related logical errors and violations of the face tracking annotation format.

[0086] Furthermore, the facial data is enhanced. To further expand the application scope of the constructed multi-face tracking dataset, a face recognition dataset can be automatically generated according to different face categories based on the constructed multi-face tracking annotation data, which can be used for training and testing face recognition algorithms and face re-identification algorithms. To further solve the problems of insufficient sample size, unbalanced sample data, and single angle and size of face images in face recognition datasets, and to further enhance the generalization ability of face recognition models and face re-identification models, data enhancement operations can be performed on all categories of face images, randomly adjusting the image's hue, hue, saturation, contrast, brightness, rotation angle, and image size, and adding directional lighting with a gradient effect to simulate face images produced in a natural environment; at the same time, some face images with occlusion effects are added to simulate face occlusion in a real environment.

[0087] Corresponding to the above method, this embodiment also proposes a multi-face tracking dataset construction system based on the Retinaface and DeepSort algorithms, which includes an automatic labeling module, an automatic cleaning module, a manual cleaning module, and a logical format verification module, wherein:

[0088] The automatic annotation module is used to build a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm, and automatically annotate multi-face tracking data.

[0089] The automatic cleaning module is used to automatically clean multi-face tracking and annotation data using a face dataset cleaning method based on the DBSCAN clustering algorithm.

[0090] The manual cleaning module is used to manually clean the multi-face tracking and annotation data based on automatic cleaning of the multi-face tracking and annotation data, mainly through manual proofreading.

[0091] The logic format verification module is used to perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

[0092] Those skilled in the art will appreciate that all or part of the steps in the above method can be performed by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a magnetic disk, or an optical disk. Alternatively, all or part of the steps in the above embodiment can be implemented using one or more integrated circuits. Accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or software functional modules. The present invention is not limited to any specific combination of hardware and software.

[0093] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The scope of protection of the present invention is not limited thereto. Although the present invention has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed by the present invention, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the scope of protection of the present invention. Therefore, the scope of protection of the present invention shall be subject to the scope of protection of the claims.< / cons> < / id> < / cons> < / id> < / cons> < / conf> < / cons> < / conf> < / cons> < / id> < / cons> < / id> < / cons> < / conf> < / cons> < / conf>

Claims

1. A method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, characterized in that: Include: Step 1: Build a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm to automatically annotate multi-face tracking data; The working process of the multi-face tracking framework is as follows: First, the DeepSort multi-target tracking algorithm is used to determine the facial motion trajectory of each object. Then, the determined facial motion trajectory ID is associated with the facial bounding boxes detected by the Retinaface face detection algorithm to automatically generate the true value box information of the face tracking data. Step 2: Use the face dataset cleaning method based on the DBSCAN clustering algorithm to automatically clean the multi-face tracking and annotation data, specifically including: Extract all face images in the video sequence based on the automatic annotation data generated by the multi-face tracking framework, and save the face images with the same face track ID in the same folder; Rename the image, and associate the new image name with the face category and face track ID; Resize all face images to the same size and perform histogram equalization on the images; The clarity of the image is evaluated by calculating the Laplace variable of the face image, and the clearest face image is used as the representative face image in the face trajectory; Use the DBSCAN clustering algorithm to cluster the representative face images in each trajectory; According to the image name, the face images that did not participate in the clustering operation are reallocated to their respective categories; According to the cleaned face categories and the name indexes of the face images in each category, the initial multi-face tracking and annotation data is automatically corrected to obtain the cleaned multi-face tracking and annotation data; Step 3: Based on the automatic cleaning of the multi-face tracking and annotation data, the multi-face tracking and annotation data is manually cleaned mainly by manual proofreading; Step 4: Perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

2. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 1, characterized in that: The Retinaface face detection model includes a feature pyramid network, a context head module, and a cascaded multi-task loss function. First, the feature pyramid network acquires the face image and outputs feature maps of different scales. Then the context head module predicts the face bounding box. Finally, the cascaded multi-task loss function is used for model training. The DeepSort multi-target tracking algorithm first uses the Retinaface algorithm to generate a face bounding box for each frame of the video. Then, the appearance features of each face target are extracted through a deep learning model. Then, the data association method is used to match the detection result of the current frame with the tracked face target of the previous frame. Finally, the tracking result of each face target in the video sequence is output.

3. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 1, characterized in that: The face bounding box information format is:<bb_left> ,<bb_top> ,<bb_right> ,<bb_bottom> , <conf> , <cons>, where represents the frame number,<bb_left> Indicates the horizontal coordinate of the upper left corner of the bounding box,<bb_top> Indicates the ordinate of the upper left corner of the bounding box,<bb_right> Indicates the horizontal coordinate of the lower right corner of the bounding box;<bb_bottom> Indicates the ordinate of the lower right corner of the bounding box, <conf>represents the confidence of the bounding box, <cons>Indicates whether to consider the bounding box information; the format of the true value box information of face tracking data is:, <id>,<bb_left>,<bb_top>,<bb_right>,<bb_bottom>, <cons>,in <id>Indicates the track ID. The face track ID number of the same object is the same; <cons> Indicates whether to consider the true value box information; the rest of the meaning is the same as explained in the above face bounding box information.< / cons> < / id> < / cons> < / id> < / cons> < / conf> < / cons> < / conf> 4. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 1, characterized in that: Manual cleaning of multi-face tracking and annotation data specifically includes: First, we manually compare videos with facial bounding boxes and trajectory IDs, record the associated indexes of the facial trajectory IDs that need to be corrected, and use script tools to automatically correct the annotated data based on the associated indexes, thereby correcting the problem of multiple IDs corresponding to the same face. Then, by manually comparing videos with face bounding boxes and trajectory IDs, we used Darklabel annotation software and script tools to correct problems such as missing faces and different faces corresponding to the same ID.

5. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 4, characterized in that: To unify the annotation format of multiple face tracking datasets, the following manual annotation rules are determined: ① Standardize the annotation range of the face bounding box; ② Define all faces that can be detected by the Retinaface algorithm or have recognizable facial features as valid faces, and select valid faces for annotation; ③ Standardize the starting and ending positions of the face track; ④ Face occlusion is defined as the face occlusion that causes the face tracking track ID to switch.

6. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 5, characterized in that: The definition of face occlusion is as follows: the face of an object has been successfully detected, but in a certain frame, the face of the object suddenly cannot be detected, and the face bounding box of the object in the previous frame is still within the monitoring field of view; another case is that the face bounding box coordinates exceed the monitoring field of view.

7. A multi-face tracking dataset construction system based on Retinaface and DeepSort algorithms, characterized by: The method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms according to claim 1 comprises an automatic labeling module, an automatic cleaning module, a manual cleaning module, and a logical format verification module, wherein: Automatic labeling module, used to build a multi-face tracking framework based on Retinaface face detection algorithm and DeepSort multi-target tracking algorithm, and automatically label multi-face tracking data; Automatic cleaning module, used to automatically clean multi-face tracking and annotation data using a face dataset cleaning method based on the DBSCAN clustering algorithm; The manual cleaning module is used to manually clean the multi-face tracking and annotation data based on the automatic cleaning of the data, mainly through manual proofreading; The logic format verification module is used to perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images