Multi-face tracking data set construction method and system based on Retinaface and DeepSort algorithms

Abstract

The invention relates to the technical field of multi-face tracking, in particular to a multi-face tracking data set construction method and system based on Retinaface and DeepSort algorithms, and the method comprises the steps: constructing a multi-face tracking framework based on a Retinaface face detection algorithm and a DeepSort multi-target tracking algorithm, and carrying out the automatic marking of multi-face tracking data; a face data set cleaning method based on a DBSCAN clustering algorithm is adopted to automatically clean the multi-face tracking annotation data; on the basis of automatic cleaning of the multi-face tracking annotation data, the multi-face tracking annotation data are cleaned manually with manual proofreading as a main component; and logic detection and format detection are carried out on the cleaned multi-face tracking annotation data, and the conditions of logic errors and inconformity with the face tracking annotation format are eliminated. According to the method, the high-quality face tracking data set can be efficiently generated.

Description

Technical Field

[0001] The present invention relates to the technical field of multi-face tracking, and in particular to a method and system for constructing a multi-face tracking data set based on Retinaface and DeepSort algorithms. Background Art

[0002] The multi-face tracking dataset is a specific research direction of the multi-target tracking dataset. It can be used to train face tracking models based on deep learning technology and provide a fair testing platform for various face tracking algorithms to compare the performance of each algorithm. Currently, due to the lack of datasets specifically for face tracking, the research and application of face tracking technology is limited.

[0003] There are two main problems in the current research on face tracking datasets. First, there are too few datasets dedicated to face tracking. At present, most tracking datasets are used in the field of single target tracking or multi-target tracking. Face tracking is only a specific research direction in the field of target tracking. There are already abundant target tracking datasets for pedestrians, vehicles, animals and other objects, while there are relatively few datasets and performance evaluation indicators for face tracking tasks. Even the Chokepoint face tracking dataset, which is widely used in the industry, only tracks the left and right eye positions of pedestrians in simple scenes (there is only one pedestrian in a single frame), but does not provide annotation data for the complex scenes "P2E_S5" and "P2L_S5" where multiple pedestrians appear at the same time and there is mutual occlusion. The face tracking dataset needs to be further improved. Second, the production cost of face tracking datasets is high, the standards are not unified, and there is a lack of systematic face tracking dataset construction methods. At present, most tracking datasets are produced manually using annotation tools. Common annotation tools include LabelImg, DarkLabel, and X-Anylabeling. The manual annotation method is costly, which greatly limits the scale of dataset production. For example, when Barquero G et al. conducted research on multi-face tracking technology, they did not find suitable multi-face tracking data, so they used a combination of face detection and manual annotation to annotate the "P2E_S5" and "P2L_S5" data in Chokepoint and some YouTube videos. However, the cost of manually associating a large number of face tracks will undoubtedly be very high; the construction time of the famous VggFace2 dataset in the field of face recognition took nearly 5 months, of which manual annotation and verification took nearly 4 months; the annotation and verification of the CPLFW dataset used for cross-age face recognition research was completed with the participation of up to 300 volunteers. On the other hand, the standards and quality of manually annotated datasets are different, and it is inevitable that there are mislabeled and missing data. For example, a study by Fei.W et al. showed that the CASIA-Webface, MS-Celeb-1M, and MegaFace datasets used for face recognition task research contain a large number of labeling errors and duplicate data. Summary of the invention

[0004] The present invention aims to solve the problems of high manual annotation cost, inconsistent annotation quality and low manual annotation efficiency of current face tracking datasets, and proposes a method and system for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, which can generate high-quality multi-face tracking datasets efficiently and at low cost.

[0005] To achieve the above purpose, the technical solution adopted is:

[0006] A method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, including:

[0007] Build a multi-face tracking framework based on Retinaface face detection algorithm and DeepSort multi-target tracking algorithm to automatically annotate multi-face tracking data;

[0008] The face data set cleaning method based on DBSCAN clustering algorithm is used to automatically clean the multi-face tracking and annotation data;

[0009] On the basis of automatic cleaning of multi-face tracking and annotation data, manual cleaning is mainly carried out on multi-face tracking and annotation data through manual proofreading;

[0010] Perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

[0011] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, the working process of the multi-face tracking framework is as follows:

[0012] Firstly, the DeepSort multi-target tracking algorithm is used to determine the face motion trajectory of each object. Then, the determined face motion trajectory ID is associated with the face bounding box detected by the Retinaface face detection algorithm to realize the automatic generation of the true value box information of face tracking data.

[0013] According to the method for constructing a multi-face tracking data set based on the Retinaface and DeepSort algorithms of the present invention, further, the Retinaface face detection model includes a feature pyramid network, a context head module and a cascaded multi-task loss function. First, the feature pyramid network obtains the face image and outputs feature maps of different scales, then the context head module predicts the face bounding box, and finally the cascaded multi-task loss function is used to train the model; the DeepSort multi-target tracking algorithm first uses the Retinaface algorithm to generate a face bounding box for each frame of the video, and then extracts the appearance features of each face target through a deep learning model, and then uses the data association method to match the detection result of the current frame with the tracked face target of the previous frame; finally, the tracking result of each face target in the video sequence is output.

[0014] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, the face bounding box information format is:<bb_left> ,<bb_top> ,<bb_right> ,<bb_bottom> , <conf> , <cons>, where represents the frame number,<bb_left> Indicates the horizontal coordinate of the upper left corner of the bounding box,<bb_top> Indicates the ordinate of the upper left corner of the bounding box,<bb_right> Indicates the horizontal coordinate of the lower right corner of the bounding box;<bb_bottom> Indicates the ordinate of the lower right corner of the bounding box, <conf>represents the confidence of the bounding box, <cons>Indicates whether to consider the detection information; the format of the face tracking data truth frame information is:, <id>,<bb_left>,<bb_top>,<bb_right>,<bb_bottom>, <cons>,in <id>Indicates the track ID. The face track ID number of the same object is the same; <cons>Indicates whether to consider the true value information; the rest of the meaning is the same as explained in the above face bounding box information.

[0015] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, a face dataset cleaning method based on the DBSCAN clustering algorithm is used to automatically clean the multi-face tracking annotation data, specifically including:

[0016] Extract all face images in the video sequence based on the automatic annotation data generated by the multi-face tracking framework, and save the face images with the same face track ID in the same folder;

[0017] Rename the image, and associate the new image name with the face category and face track ID;

[0018] Resize all face images to the same size and perform histogram equalization on the images;

[0019] The clarity of the image is evaluated by calculating the Laplace variable of the face image, and the clearest face image is used as the representative face image in the face trajectory;

[0020] Use the DBSCAN clustering algorithm to cluster the representative face images in each trajectory;

[0021] According to the image names, the face images that did not participate in the clustering operation were reallocated to their respective categories;

[0022] According to the cleaned face categories and the name indexes of face images in each category, the initial multi-face tracking and labeling data is automatically corrected to obtain the cleaned multi-face tracking and labeling data.

[0023] According to the method for constructing a multi-face tracking data set based on the Retinaface and DeepSort algorithms of the present invention, further, manually cleaning the multi-face tracking annotation data specifically includes:

[0024] First, we manually compare videos with face bounding boxes and track IDs, record the associated index of the face track ID that needs to be corrected, and use script tools to automatically correct the annotation data based on the associated index, thereby correcting the problem of multiple IDs corresponding to the same face.

[0025] Then, by manually comparing the videos with face bounding boxes and trajectory IDs, the Darklabel annotation software and script tools were used to correct the problems of missing faces and different faces corresponding to the same ID.

[0026] According to the method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms of the present invention, further, in order to unify the annotation format of the multi-face tracking dataset, the following manual annotation rules are determined: ① Standardize the annotation range of the face boundary box; ② Define all faces that can be detected by the Retinaface algorithm or whose facial features can be recognizable as valid faces, and select valid faces for annotation; ③ Standardize the starting position and ending position of the face track; ④ Face occlusion is to define the face occlusion situation that causes the face tracking track ID to switch.

[0027] According to the method for constructing a multi-face tracking dataset based on the Retinaface and DeepSort algorithms of the present invention, further, the judgment condition of face occlusion is defined as: the face of an object has been successfully detected, until a certain frame, the face of the object suddenly cannot be detected, and the face boundary box of the object in the previous frame is still within the monitoring view range; another situation is that the coordinates of the face boundary box exceed the monitoring view range.

[0028] Furthermore, the present invention also provides a multi-face tracking data set construction system based on Retinaface and DeepSort algorithms, comprising an automatic labeling module, an automatic cleaning module, a manual cleaning module and a logical format verification module, wherein:

[0029] Automatic labeling module, used to build a multi-face tracking framework based on Retinaface face detection algorithm and DeepSort multi-target tracking algorithm, and automatically label multi-face tracking data;

[0030] Automatic cleaning module, used to automatically clean multi-face tracking and annotation data using a face data set cleaning method based on the DBSCAN clustering algorithm;

[0031] The manual cleaning module is used to manually clean the multi-face tracking and annotation data based on the automatic cleaning of the multi-face tracking and annotation data, mainly through manual proofreading;

[0032] The logic format verification module is used to perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logic errors and situations that do not conform to the face tracking and annotation format.

[0033] The beneficial effects achieved by adopting the above technical solution are:

[0034] The present invention designs a multi-face tracking framework using a face detection algorithm and a multi-target tracking algorithm, and uses the framework to automatically and efficiently generate preliminarily multi-face tracking annotation data. The present invention innovatively proposes a face data set cleaning method based on the DBSCAN clustering algorithm, which greatly reduces the amount of data involved in face clustering, improves clustering efficiency while ensuring the clustering effect, and effectively identifies and merges erroneous data generated by factors such as short-term face occlusion or resolution change during the tracking process, thereby generating high-quality multi-face tracking annotation data. The present invention manually cleans the multi-face tracking annotation data automatically generated and cleaned by the multi-face tracking framework, mainly manually correcting the faces that are not detected and tracked by the multi-face tracking framework and the face trajectories that are tracked incorrectly, thereby further improving the quality of the data set. Finally, the cleaned annotation data is formatted and logically checked to eliminate logical errors and detect situations that do not conform to the annotation format of the multi-face tracking data. The present invention not only greatly reduces the workload of manual annotation of multi-face tracking data, improves the quality and efficiency of annotated data, but also further enriches the face data production method. The method can also be used to produce face recognition data and face re-identification data with a large number of pseudo-labels from video sequences, promoting the research on face recognition and face re-identification algorithms in applications such as intelligent monitoring; the method can also further expand the scope of application, and can be used to construct various types of multi-target tracking data sets by simply replacing the target detection module, providing a new idea and reference for the research on target tracking data set construction methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings of the embodiments of the present invention, wherein the drawings are only used to illustrate some embodiments of the present invention, but not to limit all embodiments of the present invention thereto.

[0036] Figure 1 It is a flowchart of a method for constructing a multi-face tracking data set based on Retinaface and DeepSort algorithms according to an embodiment of the present invention;

[0037] Figure 2 is a schematic diagram of a feature pyramid network structure according to an embodiment of the present invention;

[0038] Figure 3 1 is a schematic diagram of the composition of the cascaded multi-task loss function according to an embodiment of the present invention;

[0039] Figure 4 is a schematic diagram of the flow of the DeepSort multi-target tracking algorithm of an embodiment of the present invention;

[0040] Figure 5 is a schematic diagram of the automatic labeling process of multi-face tracking data according to an embodiment of the present invention;

[0041] Figure 6 is a schematic diagram of the automatic cleaning process of multi-face tracking and annotation data according to an embodiment of the present invention;

[0042] Figure 7 Schematic diagram of density description of DBSCAN clustering algorithm according to an embodiment of the present invention;

[0043] Figure 8 It is a schematic diagram of automatically cleaning multi-face tracking and annotation data using a clustering algorithm according to an embodiment of the present invention;

[0044] Fig. 9 is a schematic diagram of a manual cleaning process of multi-face tracking and annotation data according to an embodiment of the present invention;

[0045] Fig.10 This is an example diagram of different faces being labeled with the same ID according to an embodiment of the present invention;

[0046] Fig.11 This is an example diagram of repairing an incorrectly labeled ID during manual cleaning according to an embodiment of the present invention;

[0047] Fig.12 This is an example diagram of the same face being labeled with different IDs in an embodiment of the present invention;

[0048] Fig.13 This is an example diagram of repairing an incorrectly labeled ID during manual cleaning according to an embodiment of the present invention;

[0049] Fig.14 Schematic diagram of four types of face trajectories according to an embodiment of the present invention. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings of specific embodiments of the present invention to clearly and completely describe the exemplary scheme of the embodiment of the present invention. Unless otherwise defined, the technical terms or scientific terms used in the present invention should be the common meanings understood by people with ordinary skills in the field.

[0051] The multi-face tracking dataset proposed in this embodiment is produced by combining automatic labeling with manual labeling, which improves the efficiency of dataset production while taking into account the quality of the dataset. The basic idea of ​​automatic labeling is to automatically generate face tracking labeling data based on the face tracking algorithm. Specifically, the face detection algorithm is first used to generate face bounding box information to replace the true value box information that originally needs to be manually labeled, and then the face motion trajectory ID determined by the face tracking algorithm is used to replace the face motion trajectory ID information that originally needs to be manually labeled. The produced multi-face tracking dataset includes two parts: a training set train and a test set test. The training set contains the face bounding box information det_train and the true value box information gt_train of the objects in each video, and the test set only contains the face bounding box information det_test of the objects in the test video. This embodiment discloses a method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, such as Figure 1 As shown, the following steps are included:

[0052] Step S101: construct a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm, and automatically annotate the multi-face tracking data.

[0053] The multi-face tracking framework first uses the DeepSort multi-target tracking algorithm to determine the face motion trajectory of each object, and then associates the determined face motion trajectory ID with the face bounding box detected by the Retinaface face detection algorithm, thereby automatically generating the face tracking data truth box gt_train information. The det_traint and det_test face bounding box information are automatically generated by the Retinaface face detection algorithm.

[0054] The Retinaface face detection model includes a feature pyramid network, a context head module, and a cascaded multi-task loss function. First, the feature pyramid network obtains the input face image and outputs 5 feature maps of different scales; then, the context head module predicts the face bounding box. The first context head module predicts the face bounding box through regular anchors, while the second context head module predicts a more accurate face bounding box through regression anchors. The second context head module is generated by the first context head module. Finally, according to the context head module, Retinaface combines the loss functions of face classification, face box regression, 5 face true value coordinate regression, and 1000 3D vertex regression into a cascaded multi-task loss function for model training. The feature pyramid network structure is as follows: Figure 2 As shown, the cascaded multi-task loss function structure is as follows Figure 3 shown.

[0055] The DeepSort algorithm is a classic multi-target tracking algorithm, which is improved from the Sort algorithm. Since the Sort algorithm only considers the target motion information, it is easy to lose tracking when the target is occluded. Therefore, the DeepSort algorithm adds appearance information on the basis of motion information, that is, adds the appearance feature information of the face target extracted by the neural network to improve the tracking accuracy when the face is occluded. The operation process of the DeepSort multi-target tracking algorithm includes target detection, feature extraction, data association and state update. First, the Retinaface algorithm is used to identify the face target in each frame of the video and generate a face bounding box; then the deep learning model extracts the appearance features of each face target; then, the data association method (such as the Hungarian algorithm) is used to match the detection results of the current frame with the tracked face target of the previous frame, considering the motion model and appearance features of the face target; finally, the state of the face target is updated, the trajectory is managed, and the tracking results of each face target in the video sequence are output. The process of the DeepSort multi-target tracking algorithm is as follows Figure 4 shown.

[0056] In order to accurately describe the frame number, object ID, location and other information of the face bounding box in each video frame, a label data is set for the face image in each frame of the video. Using the above multi-face tracking technology, each face track is automatically labeled. The automatic labeling process of multi-face tracking data is as follows: Figure 5 There are two main types of annotation formats in face tracking datasets, one is face detection annotation information, and the other is the true value information of face trajectory.

[0057] The face bounding box information format is:

[0058] ,<bb_left> ,<bb_top> ,<bb_right> ,<bb_bottom> , <conf> , <cons>, where represents the frame number,<bb_left> Indicates the horizontal coordinate of the upper left corner of the bounding box,<bb_top> Indicates the ordinate of the upper left corner of the bounding box,<bb_right> Indicates the horizontal coordinate of the lower right corner of the bounding box;<bb_bottom> Indicates the ordinate of the lower right corner of the bounding box, <conf>represents the confidence of the bounding box, <cons>Indicates whether to consider this detection information.

[0059] The format of the true value frame information of face tracking data is:

[0060] , <id>,<bb_left>,<bb_top>,<bb_right>,<bb_bottom>, <cons>,in <id>Indicates the track ID. The face track ID number of the same object is the same; <cons>Indicates whether to consider the true value information; the rest of the meaning is the same as explained in the above face bounding box information.

[0061] The data cleaning of this embodiment is generally divided into two steps. First, "automatic cleaning of multi-face tracking and annotation data" is performed, and its core is to use the face clustering algorithm to clean the face images in each category generated based on the face tracking data; then "manual cleaning of multi-face tracking and annotation data" is performed, which mainly cleans the multi-face tracking data annotation files after automatic cleaning, including cleaning of missed labels, multiple labels, and mislabeled data that were not successfully corrected in the automatic cleaning stage, so as to further improve the quality of the data set.

[0062] Step S102: Automatically clean the multi-face tracking and annotation data using a face data set cleaning method based on the DBSCAN clustering algorithm.

[0063] "Automatic face data cleaning" is performed after preprocessing the face images by unifying the image size and filtering blurred images. The DBSCAN clustering algorithm is mainly used to cluster the representative faces in each track. This method conforms to the law of generating face track images by the multi-face tracking framework, that is, the face images in the same track are likely to belong to the same person. Even if there are multiple faces in the same track, they can be corrected in the manual cleaning stage. Therefore, during automatic cleaning, only face images of different tracks need to be clustered, thereby reducing the number of face images involved in clustering and improving computational efficiency and clustering quality. According to the face categories after face clustering cleaning and the face image name index in each category, the labeled data generated by the multi-face tracking framework is automatically corrected. The automatic cleaning process of multi-face tracking labeled data is as follows: Figure 6 The specific operation process is as follows:

[0064] (1) Extract all face images in the video sequence based on the automatic annotation data generated by the multi-face tracking framework, and save the face images with the same face track ID in the same folder.

[0065] (2) Rename the image and associate the new image name with the face category and face track ID to facilitate subsequent classification.

[0066] (3) All face images are resized to the same size to eliminate the impact of size differences on clustering results.

[0067] (4) Perform histogram equalization on the image to enhance the contrast of the image and reduce the brightness difference between different images.

[0068] (5) The clarity of the image is evaluated by calculating the Laplace variable of the face image. The unclear face image is deleted through a custom threshold, and the clearest face image is copied. This face image is called the representative face data in the face track.

[0069] (6) The DBSCAN clustering algorithm is used to cluster the representative face images in each trajectory, and further correctly cluster the wrong category face images generated by the multi-face tracking framework.

[0070] (7) According to the image names, the face images that do not participate in the clustering operation are reallocated to their respective categories.

[0071] (8) Based on the cleaned face categories and the name index of the face images in each category, the initial multi-face tracking and annotation data is automatically corrected using the written script to obtain the cleaned multi-face tracking and annotation data.

[0072] In the process of automatic face data cleaning, the data generated by the multi-target tracking algorithm has the characteristics of continuity in time and space. Considering that the face images in the same face track belong to the same person with a high probability, the face images in different face tracks in the same frame must not belong to the same person. Therefore, when performing face clustering operations, only one representative face data is taken from each face track of the automatically generated data for face clustering. While greatly reducing the amount of calculation of the clustering algorithm, it also reduces the impact of the deterioration of clustering effect caused by the increase in density of a large amount of face data in the feature space. In this case, the DBSCAN clustering algorithm is selected to cluster face data to achieve the purpose of data cleaning. A significant advantage of the DBSCAN clustering algorithm is that it does not require the user to pre-set the number of clusters. This flexibility makes it perform well in the face of an unknown number of clusters. In addition, DBSCAN has the ability to identify and process noise and outliers, which is particularly critical in face image clustering because image data often contains multiple interference elements. Although the parameter setting of DBSCAN has some influence on the final clustering effect, its design allows users to adjust the parameters according to the specific data set and clustering target, thus providing greater adaptability. Finally, DBSCAN can also effectively process high-dimensional data, which is an indispensable advantage for the task of clustering face images containing a large amount of feature information. The density description diagram of the DBSCAN clustering algorithm is shown in the figure below. Figure 7 As shown in the figure, the clustering algorithm automatically cleans the multi-face tracking and annotation data. Figure 8 shown.

[0073] Step S103: Based on the automatic cleaning of the multi-face tracking and annotation data, the multi-face tracking and annotation data is manually cleaned mainly by manual proofreading and supplemented by script tools. It mainly processes the face track ID mislabeling, missing labeling, and multiple labeling that are not successfully corrected in the automatic cleaning stage. The manual cleaning process of multi-face tracking and annotation data is as follows: Fig. 9 The manual cleaning process is as follows:

[0074] (1) By manually comparing videos with face bounding boxes and trajectory IDs, the associated index of the face trajectory ID that needs to be corrected is recorded, and the script tool is used to automatically correct the labeled data based on the associated index to correct the problem of multiple IDs corresponding to the same face.

[0075] (2) By manually comparing videos with face bounding boxes and trajectory IDs, the Darklabel annotation software and script tools are used to correct the problems of missing faces and different faces corresponding to the same ID.

[0076] Through further manual cleaning and detection, more accurate multi-face tracking and labeling data can be obtained. The following specifically introduces the process of manually processing wrong face data labels, missing face track ID labels, and multiple face track ID labels.

[0077] ① Mislabeling of face data. This case divides mislabeling of face data into two categories: mislabeling of face track ID and mislabeling of face detection frame. There are two types of mislabeling of face track ID. The first type refers to the fact that the same ID is assigned to different people one after another; the other type refers to the fact that different IDs are assigned to the same person one after another. Fig.10 As shown in the figure, ID 28 is successively assigned to two different face targets. During the manual labeling process, if the target with the larger frame number to which the face ID is incorrectly assigned is the first labeled ID, the incorrect ID will be corrected to the next digit of the last ID number in the current labeled data. For example, if the last ID in the current labeled data is 96, then the "manual labeling auxiliary script" can be used to correct all incorrect IDs to 97. Fig.11 As shown in the figure, new annotated data is obtained. The next time you manually annotate, the last ID is found based on the latest annotated data and the wrong ID is modified, and so on. The second case of wrong face track ID is as follows: Fig.12 As shown in the figure, when the same face target is labeled with different IDs, the corresponding correct and incorrect IDs in the frames are manually recorded, and then the "face tracking and labeling data correction script" is used to automatically correct the face track IDs after correction. Fig.13 As shown. Face detection frame mislabeling means that due to various factors, the face detection algorithm cannot correctly perform face detection, and the face bounding box has obvious deviations. At this time, Darklabel annotation software can be used for auxiliary annotation and generate face bounding box data. According to the workload required for correction, choose manual or script-assisted methods to add the corrected data into the annotation data file.

[0078] ② Missing face track ID. The face track missing in this case can be divided into two situations. One is that the face detection algorithm did not correctly detect the face in the video, resulting in the failure to track the face; the other is that some low-quality face data was filtered out during the automatic cleaning phase to improve clustering quality and efficiency. For the first situation, Darklabel software can be used to follow the manual labeling rules for labeling; for the second situation, refer to the face track ID mislabeling situation in ①, and use the "Face Tracking Labeling Data Correction Script" and "Manual Labeling Auxiliary Script" to correct it.

[0079] ③ Over-labeling of face track IDs. Over-labeling of face track IDs in this case refers to when face images appear on reflective surfaces such as glass and water, which results in the labeling of face tracks that do not need to be labeled. At this time, the redundant face track IDs can be deleted manually or with the help of auxiliary scripts.

[0080] In order to further standardize the manual cleaning standards, the following manual labeling rules have been determined.

[0081] ① Determination of the face bounding box. In order to unify the annotation rules, this case standardizes the annotation range of the face bounding box, requiring the face bounding box to be aligned with the range of the face as accurately as possible. The bounding box should contain all pixels of the left and right eyes, nose, and left and right corners of the mouth. For a front view and unobstructed face, the tip of the nose is approximately located at the center of mass of the face bounding box, and the left and right side bounding boxes are between the corners of the eyes and the ears. The upper bounding box and the horizontal line passing through the tip of the nose are roughly symmetrical about the horizontal line between the eyebrows. The lower bounding box is flush with the lower jaw of the face, and the aspect ratio of the face bounding box is about 3:4; for a side view or an obstructed face, at least one eye, nose tip and one corner of the mouth should be ensured, otherwise it is regarded as an invalid face image and the face bounding box is not annotated. The upper and lower bounding boxes are consistent with the rules in the front view.

[0082] ② Valid face conditions. All faces that can be detected by the Retinaface algorithm or whose facial features can be recognized are called valid faces. In order to ensure that various face tracking algorithms can give full play to their algorithm performance when using this dataset for evaluation, all valid faces are labeled. When evaluating the performance of face tracking algorithms, each algorithm can choose a suitable face size for evaluation based on its own characteristics. As long as each algorithm is evaluated under the same conditions, the final results are comparable.

[0083] ③ The start and end of the track. The starting position of the face track is usually determined by the face detection algorithm. When the face of an object is detected for the first time, the corresponding frame is considered to be the starting position of the face track of the object. If there is a small face that is not detected by the face detection algorithm and meets the above-mentioned "valid face condition" standard, the face in the frame is manually marked as the starting position of the track; in order to be applicable to the evaluation of face tracking algorithms with face re-identification functions, when the face is blocked or the face frame coordinates exceed the monitoring field of view, the track ID is always kept unchanged to prevent the face corresponding to the ID from appearing again until the last frame of the video sequence. Therefore, the end position of a track is the frame corresponding to the last successful detection in the track.

[0084] ④ Face occlusion. Here, "face occlusion" is defined as the face occlusion that causes the face tracking trajectory ID to switch, and does not include occlusions that have never been detected by the face detection algorithm. Judgment conditions for face occlusion: the face of an object has been successfully detected, until a certain frame, the face of the object suddenly cannot be detected, and the face frame of the object in the previous frame is still within the entire monitoring view range; another special case of occlusion, that is, the face frame coordinates exceed the monitoring view range, and its judgment condition is: any vertex coordinate of the bounding box is negative or the horizontal coordinate is greater than the maximum horizontal coordinate of the monitoring view, or the vertical coordinate is greater than the maximum vertical coordinate of the monitoring view. When face occlusion occurs, it is easy to cause face ID switching or trajectory interruption, resulting in track fragmentation, such as Fig.14 shown.

[0085] Step S104, multi-face tracking annotation data logic and format verification. Since there may be human errors in the manual cleaning stage, resulting in a small amount of label data format errors, or "multiple identical face tracking frames in the same video frame" and "multiple identical track IDs in the same video frame" logical errors, it is necessary to use the "annotation data detection script" provided in this case to perform format and logic detection on the cleaned annotation data to eliminate related logical errors and situations that do not conform to the face tracking annotation format.

[0086] Furthermore, the face data is enhanced. To further expand the application scope of the constructed multi-face tracking dataset, a face recognition dataset can be automatically generated according to different face categories based on the constructed multi-face tracking annotation data, which can be used for training and testing face recognition algorithms and face re-identification algorithms. To further solve the problems of insufficient sample size, unbalanced sample data, and single angle and size of face images in face recognition datasets, and to further enhance the generalization ability of face recognition models and face re-identification models, data enhancement operations can be performed on all categories of face images, randomly adjusting the image's hue, hue, saturation, contrast, brightness, rotation angle, and image size, and adding directional lighting with a gradient effect to simulate face images produced in a natural environment; at the same time, some face images with occlusion effects are added to simulate face occlusion in a real environment.

[0087] Corresponding to the above method, this embodiment also proposes a multi-face tracking dataset construction system based on Retinaface and DeepSort algorithms, which includes an automatic labeling module, an automatic cleaning module, a manual cleaning module and a logical format verification module, wherein:

[0088] The automatic labeling module is used to build a multi-face tracking framework based on the Retinaface face detection algorithm and the DeepSort multi-target tracking algorithm, and automatically label the multi-face tracking data.

[0089] The automatic cleaning module is used to automatically clean the multi-face tracking and annotation data using a face data set cleaning method based on the DBSCAN clustering algorithm.

[0090] The manual cleaning module is used to manually clean the multi-face tracking and annotation data based on automatic cleaning of the multi-face tracking and annotation data, mainly through manual proofreading.

[0091] The logic format verification module is used to perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logic errors and situations that do not conform to the face tracking and annotation format.

[0092] Those skilled in the art will appreciate that all or part of the steps in the above method can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, such as a read-only memory, a disk or an optical disk. Optionally, all or part of the steps in the above embodiment can also be implemented using one or more integrated circuits, and accordingly, each module / unit in the above embodiment can be implemented in the form of hardware or in the form of software function modules. The present invention is not limited to any specific form of combination of hardware and software.

[0093] Finally, it should be noted that the above-described embodiments are only specific implementations of the present invention, which are used to illustrate the technical solutions of the present invention, rather than to limit them. The protection scope of the present invention is not limited thereto. Although the present invention is described in detail with reference to the above-described embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-described embodiments within the technical scope disclosed by the present invention, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.< / cons> < / id> < / cons> < / id> < / cons> < / conf> < / cons> < / conf> < / cons> < / id> < / cons> < / id> < / cons> < / conf> < / cons> < / conf>

Claims

1. A method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms, characterized in that: Include: Build a multi-face tracking framework based on Retinaface face detection algorithm and DeepSort multi-target tracking algorithm to automatically annotate multi-face tracking data; The face data set cleaning method based on DBSCAN clustering algorithm is used to automatically clean the multi-face tracking and annotation data; On the basis of automatic cleaning of multi-face tracking and annotation data, manual cleaning is mainly carried out on multi-face tracking and annotation data through manual proofreading; Perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logical errors and situations that do not conform to the face tracking and annotation format.

2. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 1, characterized in that: The working process of the multi-face tracking framework is: Firstly, the DeepSort multi-target tracking algorithm is used to determine the face motion trajectory of each object. Then, the determined face motion trajectory ID is associated with the face bounding box detected by the Retinaface face detection algorithm to realize the automatic generation of the true value box information of face tracking data.

3. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 2, characterized in that: The Retinaface face detection model includes a feature pyramid network, a context head module and a cascaded multi-task loss function. First, the feature pyramid network obtains the face image and outputs feature maps of different scales. Then the context head module predicts the face bounding box. Finally, the cascaded multi-task loss function is used for model training. The DeepSort multi-target tracking algorithm first uses the Retinaface algorithm to generate a face bounding box for each frame of the video, and then extracts the appearance features of each face target through a deep learning model. Then, the data association method is used to match the detection result of the current frame with the tracked face target of the previous frame. Finally, the tracking result of each face target in the video sequence is output.

4. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 2, characterized in that: The face bounding box information format is:<bb_left> ,<bb_top> ,<bb_right> ,<bb_bottom> , <conf> , <cons>, where represents the frame number,<bb_left> Indicates the horizontal coordinate of the upper left corner of the bounding box,<bb_top> Indicates the ordinate of the upper left corner of the bounding box,<bb_right> Indicates the horizontal coordinate of the lower right corner of the bounding box;<bb_bottom> Indicates the ordinate of the lower right corner of the bounding box, <conf>represents the confidence of the bounding box, <cons>Indicates whether to consider the detection information; the format of the true value frame information of face tracking data is:, <id>,<bb_left>,<bb_top>,<bb_right>,<bb_bottom>, <cons>,in <id>Indicates the track ID. The face track ID number of the same object is the same; <cons> Indicates whether to consider the true value information; the rest of the meaning is the same as explained in the above face bounding box information.< / cons> < / id> < / cons> < / id> < / cons> < / conf> < / cons> < / conf> 5. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 1, characterized in that: The face data set cleaning method based on DBSCAN clustering algorithm is used to automatically clean the multi-face tracking and annotation data, specifically including: Extract all face images in the video sequence based on the automatic annotation data generated by the multi-face tracking framework, and save the face images with the same face track ID in the same folder; Rename the image, and associate the new image name with the face category and face track ID; Resize all face images to the same size and perform histogram equalization on the images; The clarity of the image is evaluated by calculating the Laplace variable of the face image, and the clearest face image is used as the representative face image in the face trajectory; Use the DBSCAN clustering algorithm to cluster the representative face images in each trajectory; According to the image names, the face images that did not participate in the clustering operation were reallocated to their respective categories; According to the cleaned face categories and the name indexes of face images in each category, the initial multi-face tracking and annotation data is automatically corrected to obtain the cleaned multi-face tracking and annotation data.

6. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 1, characterized in that: Manual cleaning of multi-face tracking and annotation data specifically includes: First, we manually compare videos with face bounding boxes and track IDs, record the associated index of the face track ID that needs to be corrected, and use script tools to automatically correct the annotation data based on the associated index, thereby correcting the problem of multiple IDs corresponding to the same face. Then, by manually comparing the videos with face bounding boxes and trajectory IDs, the Darklabel annotation software and script tools were used to correct the problems of missing faces and different faces corresponding to the same ID.

7. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 6, characterized in that: In order to unify the annotation format of multiple face tracking datasets, the following manual annotation rules are determined: ① Standardize the annotation range of the face bounding box; ② Define all faces that can be detected by the Retinaface algorithm or have recognizable facial features as valid faces, and select valid faces for annotation; ③ Standardize the starting and ending positions of the face track; ④ Face occlusion is to define the face occlusion that causes the face tracking track ID to switch.

8. The method for constructing a multi-face tracking dataset based on Retinaface and DeepSort algorithms according to claim 7, characterized in that: The judgment condition for face occlusion is defined as follows: the face of an object has been successfully detected, but in a certain frame, the face of the object suddenly cannot be detected, and the face bounding box of the object in the previous frame is still within the monitoring view range; another case is that the coordinates of the face bounding box exceed the monitoring view range.

9. A multi-face tracking dataset construction system based on Retinaface and DeepSort algorithms, characterized in that: It includes automatic labeling module, automatic cleaning module, manual cleaning module and logical format verification module, among which: Automatic labeling module, used to build a multi-face tracking framework based on Retinaface face detection algorithm and DeepSort multi-target tracking algorithm, and automatically label multi-face tracking data; Automatic cleaning module, used to automatically clean multi-face tracking and annotation data using a face data set cleaning method based on the DBSCAN clustering algorithm; The manual cleaning module is used to manually clean the multi-face tracking and annotation data based on the automatic cleaning of the multi-face tracking and annotation data, mainly through manual proofreading; The logic format verification module is used to perform logic detection and format detection on the cleaned multi-face tracking and annotation data to eliminate logic errors and situations that do not conform to the face tracking and annotation format.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 8 are implemented.

Images