An open set-oriented large-scale face retrieval dataset cleaning method and system

Abstract

The application discloses a large-scale face retrieval data set cleaning method and system for an open set, and the cleaning method comprises seven steps of face preprocessing, face detection and key point detection, face classification, face feature extraction, face similarity search, face data merging and face data deletion. The method is based on a multi-threshold, multi-path parallel and multi-round iteration method based on similarity search, and can automatically, accurately and efficiently complete the whole-process cleaning work of large-scale face data without manual participation. The search indexer based on the similarity search adopts an incremental construction method, and the face feature vectors of the same identity ID are only one in the search indexer, so that the face data of the same identity ID can be efficiently placed in the same folder.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular discloses a method and system for cleaning large-scale face retrieval datasets for open sets. Background Technology

[0002] With continuous breakthroughs in deep learning technology and increasing computing power, the field of computer vision has made groundbreaking progress. Face retrieval, as a key research direction in computer vision, is widely used in security, social networking, and everyday entertainment. However, current face retrieval still faces challenges such as noisy and small-scale publicly available training datasets, especially the scarcity of Asian faces. Currently, the largest and cleanest publicly available face training dataset is Glint360K, which contains only 360,000 IDs and 17 million images. Furthermore, models trained on this dataset still exhibit very poor accuracy in retrieving Asian faces, far from meeting the data requirements of large-scale face retrieval systems for open datasets.

[0003] However, relying entirely on manual cleaning of massive amounts of facial data presents several problems, including but not limited to the following: 1) data security issues; 2) inability to accurately classify massive amounts of facial images by identity ID; 3) time-consuming and labor-intensive.

[0004] After obtaining images containing human faces through the internet or other means, how can we efficiently create a large-scale and clean human face dataset that can be used in real-world scenarios?

[0005] Patent 201810930672.0 discloses a method and mechanism for cleaning facial data. The method involves acquiring first facial data related to a target person, extracting facial features from each facial data set based on a first facial recognition model, and determining the first facial feature of the target person. Based on the facial features of each facial data set and the first facial feature of the target person, the method filters out the first facial data set that does not need cleaning, the second facial data set that needs to be deleted, and the third facial data set that needs cleaning. The third facial data set that needs cleaning is manually cleaned. This invention can reduce the labor costs and expenses of cleaning, but it cannot automate the entire process and still requires manual intervention. Furthermore, the claims and embodiments do not show the applicable scale for data cleaning tasks; therefore, its applicability at scales of tens of millions or even hundreds of millions remains to be verified.

[0006] Patent 202010288271.7 discloses a face clustering method, apparatus, electronic device, and computer-readable storage medium. The method involves identifying each face data point in a face dataset as a node; calculating the similarity between each pair of nodes and constructing edges between nodes with similarity greater than a specified threshold; constructing a data graph based on the edges and their initial weights; and determining the target category of each node in the data graph based on the initial weights and initial category identifiers, according to a preset number of iterations, to complete the clustering of face data in the face dataset. This invention effectively alleviates the problem of single-class anomalies and improves the accuracy of clustering. However, given the characteristics of clustering, it is currently difficult to apply accurately and efficiently to large-scale face data cleaning problems.

[0007] Patent CN202010188856.1 discloses a method, apparatus, and device for cleaning facial data. The method includes: acquiring a facial dataset within a preset time period; determining the similarity value between each pair of faces in the dataset; identifying duplicate facial data based on the similarity values; and selectively retaining one of the duplicate facial data. This invention improves the efficiency of data storage and processing, but it only addresses the deduplication problem in facial data cleaning and does not address the core issue of facial data cleaning: how to efficiently and accurately group large-scale facial data according to the same facial identity ID.

[0008] Patent CN202111061863.6 discloses a two-stage noise cleaning method for large-scale face datasets. This method utilizes a two-stage strategy, allowing the network to spontaneously detect closed-set and open-set noise samples during training. Closed-set noise samples are reused, while open-set noise samples are removed from the training set. This invention solves the problem of cleaning a basically usable closed-set face dataset with only slight noise. However, in real-world scenarios, face data is a randomly distributed open set, making data cleaning far more difficult than cleaning a closed-set dataset as described in this invention. Furthermore, the embodiments only use a relatively small closed-set dataset (weface) for illustration; the applicability of this cleaning scheme to larger datasets remains to be verified.

[0009] The methods and embodiments disclosed in the aforementioned patents primarily address the problem of cleaning closed-set face datasets with a defined number of categories, and these methods generally require manual intervention. This invention proposes a cleaning method for large-scale open-set face datasets with an uncertain number of categories. It employs a multi-threshold, multi-path parallel, and multi-round iterative approach based on similarity search to automatically, accurately, and efficiently complete the entire cleaning process for large-scale face data, eliminating the need for manual intervention. Furthermore, this invention's cleaning method is applicable to different data volumes, meeting the cleaning needs of datasets in the millions, tens of millions, or even hundreds of millions.

[0010] Based on the above-mentioned existing patents and related technologies, the existing patents currently have the following problems:

[0011] Existing patents fail to address the problem of cleaning large-scale open-set face data with an uncertain number of categories required for real-world scenarios. They primarily address the cleaning problem of closed-set face datasets with a definite number of categories. For example, noisy face datasets obtained by crawling the internet using identity IDs and other information have a definite number of categories, and the face images in each category have a certain degree of accuracy, containing only some erroneous faces or non-face noise data.

[0012] For large-scale datasets (such as tens of millions or even hundreds of millions of faces), clustering algorithms struggle to converge, have low accuracy, and require significant amounts of memory and time. Furthermore, clustering algorithms generally require setting an initial number of clusters, making them unsuitable for open-set face datasets with an uncertain number of clusters.

[0013] To address the aforementioned issues, this invention proposes a face data cleaning method and system based on similarity search, featuring multiple thresholds, multi-path parallelism, and multi-round iteration, without requiring manual intervention, for large-scale face data in open sets of real-world application scenarios. Summary of the Invention

[0014] This invention provides a method and system for cleaning large-scale face retrieval datasets for open sets, aiming to address the technical problem of cleaning large-scale open set face data where the number of categories required for real-world scenarios is uncertain, which has not been addressed in existing technologies.

[0015] One aspect of this invention relates to a method for cleaning large-scale face retrieval datasets for open sets, comprising the following steps:

[0016] The face preprocessing step involves using image processing tools to stitch together all images containing faces onto a canvas of a specified size to obtain a stitched face image.

[0017] The face detection and key point detection steps involve performing face detection and key point detection on each stitched face image to obtain several face images and key points; then, using each key point, an affine transformation is performed on the corresponding face image to obtain each aligned face image.

[0018] The face classification step involves classifying each aligned face image into usable faces and unusable faces, and then removing unusable faces.

[0019] The facial feature extraction step involves extracting facial features from each available face, obtaining facial feature vectors, and then using data analysis tools to save each facial feature vector to the disk.

[0020] The face similarity search step is completed in parallel by several face similarity sub-searches. At the same time, the face feature vector is divided into several corresponding parts. Each of the several face similarity sub-searches is responsible for the similarity search task of one face feature vector and obtains a different identity ID folder. The different identity ID folders obtained by the several face similarity sub-searches are placed in the same folder to obtain the initial face folder.

[0021] The face data merging process involves: randomly selecting several face images from various subfolders of the initial face folder; extracting face features from all randomly selected face images to obtain face feature vectors; concatenating face feature vectors with the same identity ID to obtain concatenated face feature vectors; using data analysis tools, saving all concatenated face feature vectors to disk; calculating the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the merging indexer; merging folders with similar distances less than the merging threshold to obtain a temporary merged face folder; updating the merging threshold; and continuing until the number of temporary merged face folders does not decrease significantly or the merging rounds stop at a specified number, resulting in a final merged face folder.

[0022] The steps for deleting face data are as follows: First, randomly select several face images from each subfolder of the merged face folder. Then, extract face features from all randomly selected face images to obtain face feature vectors. Concatenate the face feature vectors of individuals with the same ID to obtain concatenated face feature vectors. Using data analysis tools, save all concatenated face feature vectors to disk. Next, insert all concatenated face feature vectors into the deletion indexer and update the indexer. Finally, calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the deletion indexer to obtain the minimum similarity distance and corresponding index number. Then, store the minimum similarity distance, corresponding index number, and... The corresponding file paths are saved to the file to be deleted. Based on the file to be deleted, all folders with fewer than the deletion threshold and not the most images in the folder are deleted to obtain a temporary folder of deleted faces. The deletion threshold is updated according to a certain decay rate until the number of temporary folders of deleted faces basically does not decrease or stops after a specified number of rounds. The deleted face folder is then obtained. The network parameters for face feature extraction are trained using the deleted face folder to obtain a better face feature extraction model. The face feature extraction is further updated until the face feature extraction model reaches the desired accuracy or stops after a specified number of rounds. The deleted face folder in the last round is the final cleaned face folder.

[0023] Furthermore, the facial similarity search steps include:

[0024] Initialize the search indexer;

[0025] Insert the first face feature vector into the search indexer, update the search indexer, and save the available faces corresponding to the first face feature vector to the specified folder;

[0026] Each face feature vector is sequentially compared with all face feature vectors in the search indexer to calculate the similarity distance, obtaining the minimum similarity distance and the corresponding index number. If the minimum similarity distance is less than the set micro threshold, no operation is performed. If the minimum similarity distance is greater than the set micro threshold but less than the set small threshold, the available faces corresponding to the face feature vector are saved to the folder with the same identity ID according to the index number. If the minimum similarity distance is greater than the small threshold, there is no face feature vector with the same identity ID as the face feature vector in the search indexer. The face feature vector is then inserted into the search indexer, the search indexer is updated, and the available faces corresponding to the face feature vector are saved to another specified folder.

[0027] Obtain different ID folders for saving face images grouped by ID.

[0028] Furthermore, the facial data merging steps include:

[0029] Randomly select several face images from each subfolder of the initial face folder;

[0030] Facial features are extracted from all randomly selected facial images to obtain facial feature vectors. Facial feature vectors of the same identity ID are then concatenated to obtain concatenated facial feature vectors. Using data analysis tools, all concatenated facial feature vectors are saved to disk.

[0031] Initialize the merge indexer;

[0032] Insert all the stitched face feature vectors into the merge indexer and update the merge indexer;

[0033] Calculate the similarity distance between each stitched face feature vector and all stitched face feature vectors in the merging indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be merged;

[0034] Based on the files to be merged, folders smaller than the merging threshold are merged to obtain a temporary folder of merged faces, and the merging threshold is updated according to a certain decay rate.

[0035] Repeat all the above operations until the number of merged face temporary folders does not decrease significantly or the specified number of rounds is reached, and then obtain the merged face folders.

[0036] Furthermore, the steps for deleting facial data include:

[0037] Randomly select several face images from each subfolder of the merged face folder;

[0038] Facial features are extracted from all randomly selected facial images to obtain facial feature vectors. Facial feature vectors of the same identity ID are then concatenated to obtain concatenated facial feature vectors. Using data analysis tools, all concatenated facial feature vectors are saved to disk.

[0039] Initialize the delete indexer;

[0040] Insert all the stitched face feature vectors into the delete indexer and update the delete indexer;

[0041] Calculate the similarity distance between each stitched face feature vector and all stitched face feature vectors in the deletion indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be deleted;

[0042] Based on the files to be deleted, delete all folders that are less than the deletion threshold and do not contain the most images, obtain a temporary folder of deleted faces, and update the deletion threshold according to a certain decay rate.

[0043] Repeat all the above operations until the number of deleted face temporary folders does not decrease significantly or reaches the specified number of rounds, and then obtain the deleted face folder.

[0044] Furthermore, the merging threshold is calculated using the following formula:

[0045] Merging threshold = Initial threshold - Decay rate * Number of merging rounds

[0046] The merge threshold is used to merge different folders with the same identity ID;

[0047] The deletion threshold is calculated using the following formula:

[0048] Deletion threshold = Initial threshold - Decay rate * Number of deletion rounds

[0049] The deletion threshold is used to delete all folders whose images are less than the deletion threshold and which do not contain the most images.

[0050] Another aspect of the present invention relates to a system for cleaning large-scale face retrieval datasets for open sets, comprising:

[0051] The face preprocessing module is used to stitch together all images containing faces according to a specified size canvas using image processing tools to obtain a stitched face image.

[0052] The face detection and key point detection module is used to perform face detection and key point detection on each stitched face image to obtain several face images and key points; and to perform affine transformation on the corresponding face images using each key point to obtain each aligned face image.

[0053] The face classification module is used to classify each aligned face image into usable faces and unusable faces, and remove unusable faces;

[0054] The face feature extraction module is used to extract face features from each available face, obtain face feature vectors, and save each face feature vector to the disk in conjunction with data analysis tools;

[0055] The face similarity search module is used to complete the task in parallel by several face similarity sub-searches. At the same time, the face feature vector is divided into several corresponding parts. Each of the several face similarity sub-searches is responsible for the similarity search task of one face feature vector and obtains a different identity ID folder. The different identity ID folders obtained by the several face similarity sub-searches are placed in the same folder to obtain the initial face folder.

[0056] The face data merging module is used to randomly select several face images from various subfolders of the initial face folder; extract face features from all randomly selected face images to obtain face feature vectors; concatenate the face feature vectors of the same identity ID to obtain concatenated face feature vectors; save all concatenated face feature vectors to disk using data analysis tools; calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the merging indexer; merge folders with similar distances less than the merging threshold to obtain a merged face temporary folder; update the merging threshold until the number of merged face temporary folders does not decrease significantly or stops after a specified number of merging rounds; and obtain the merged face folder.

[0057] The face data deletion module is used to randomly select a number of face images from various subfolders of the merged face folder; extract face features from all randomly selected face images to obtain face feature vectors; concatenate the face feature vectors of people with the same identity ID to obtain concatenated face feature vectors; and save all concatenated face feature vectors to disk using data analysis tools; insert all concatenated face feature vectors into the deletion indexer and update the deletion indexer; calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the deletion indexer to obtain the minimum similarity distance and corresponding index number, and then set the minimum similarity distance and corresponding index number as... The corresponding file paths are saved to the file to be deleted. Based on the file to be deleted, all folders with fewer than the deletion threshold and not the most images in the folder are deleted to obtain a temporary folder of deleted faces. The deletion threshold is updated according to a certain decay rate until the number of temporary folders of deleted faces basically does not decrease or stops after a specified number of rounds. The deleted face folder is then obtained. The network parameters for face feature extraction are trained using the deleted face folder to obtain a better face feature extraction model. The face feature extraction is further updated until the face feature extraction model reaches the expected accuracy or stops after a specified number of rounds. The deleted face folder in the last round is the final cleaned face folder.

[0058] Furthermore, the face similarity search module includes:

[0059] The first initialization unit is used to initialize the search indexer;

[0060] The first update unit is used to insert the first face feature vector into the search indexer, update the search indexer, and save the available faces corresponding to the first face feature vector to the specified folder.

[0061] The comparison unit is used to calculate the similarity distance between each face feature vector and all face feature vectors in the search indexer, obtaining the minimum similarity distance and the corresponding index number. If the minimum similarity distance is less than a set micro threshold, no operation is performed. If the minimum similarity distance is greater than the set micro threshold but less than the set small threshold, the available faces corresponding to the face feature vector are saved to the folder with the same identity ID according to the index number. If the minimum similarity distance is greater than the small threshold, there is no face feature vector with the same identity ID as the face feature vector in the search indexer. The face feature vector is inserted into the search indexer, the search indexer is updated, and the available faces corresponding to the face feature vector are saved to another specified folder.

[0062] The first acquisition unit is used to obtain different identity ID folders that store face images grouped by identity ID.

[0063] Furthermore, the face data merging module includes:

[0064] The first extraction unit is used to randomly extract several face images from each subfolder of the initial face folder;

[0065] The stitching unit is used to extract facial features from all randomly selected face images to obtain facial feature vectors, and to stitch together the facial feature vectors of people with the same identity ID to obtain stitched facial feature vectors; combined with data analysis tools, all stitched facial feature vectors are saved to disk;

[0066] The second initialization unit is used to initialize the merge indexer;

[0067] The second update unit is used to insert all the spliced ​​face feature vectors into the merge indexer and update the merge indexer;

[0068] The second acquisition unit is used to calculate the similarity distance between each spliced ​​face feature vector and all spliced ​​face feature vectors in the merging indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be merged.

[0069] The merging unit is used to merge folders smaller than the merging threshold based on the files to be merged, obtain a temporary folder of merged faces, and update the merging threshold according to a certain decay rate.

[0070] The third acquisition unit is used to repeat all the above operations until the number of merged face temporary folders does not decrease significantly or stops after a specified number of rounds, and obtains the merged face folders.

[0071] Furthermore, the face data deletion module includes:

[0072] The second extraction unit is used to randomly extract several face images from each subfolder of the merged face folder;

[0073] The fourth acquisition unit is used to extract facial features from all randomly selected facial images to obtain facial feature vectors, and to concatenate facial feature vectors of people with the same identity ID to obtain concatenated facial feature vectors; combined with data analysis tools, all concatenated facial feature vectors are saved to disk;

[0074] The third initialization unit is used to initialize the delete indexer;

[0075] The third update unit is used to insert all the spliced ​​face feature vectors into the delete indexer and update the delete indexer;

[0076] The fifth acquisition unit is used to calculate the similarity distance between each spliced ​​face feature vector and all spliced ​​face feature vectors in the deletion indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be deleted;

[0077] The sixth acquisition unit is used to delete all folders with fewer than the deletion threshold and not the most images in the folder, based on the files to be deleted, to obtain the temporary folder of deleted faces, and update the deletion threshold according to a certain decay rate;

[0078] The seventh acquisition unit is used to repeat all the above operations until the number of deleted face temporary folders does not decrease significantly or stops after a specified number of rounds, and obtains the deleted face folders.

[0079] Furthermore, the merging threshold is calculated using the following formula:

[0080] Merging threshold = Initial threshold - Decay rate * Number of merging rounds

[0081] The merge threshold is used to merge different folders with the same identity ID;

[0082] The deletion threshold is calculated using the following formula:

[0083] Deletion threshold = Initial threshold - Decay rate * Number of deletion rounds

[0084] The deletion threshold is used to delete all folders whose images are less than the deletion threshold and which do not contain the most images.

[0085] The beneficial effects achieved by this invention are as follows:

[0086] This invention provides a method and system for cleaning large-scale face retrieval datasets for open sets. The cleaning method includes seven steps: face preprocessing, face detection and key point detection, face classification, face feature extraction, face similarity search, face data merging, and face data deletion. It adopts a multi-threshold, multi-path parallel, and multi-round iterative method based on similarity search to automatically, accurately, and efficiently complete the entire process of cleaning large-scale face data without manual intervention. The search indexer based on similarity search adopts an incremental construction method, ensuring that the face feature vector of the same identity ID exists only once in the search indexer. This efficiently enables the placement of face data with the same identity ID in the same folder and face data with different identity IDs in different folders, while simultaneously removing face data that is essentially the same or has extremely high similarity. The cleaning method is suitable for different scales of data volume and can meet the cleaning needs of millions, tens of millions, or even hundreds of millions of data. Attached Figure Description

[0087] Figure 1 This is a flowchart illustrating an embodiment of the large-scale face retrieval dataset cleaning method for open sets provided by the present invention.

[0088] Figure 2 This is a flowchart illustrating an embodiment of the face similarity search step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention.

[0089] Figure 3 This is a flowchart illustrating an embodiment of the face similarity sub-search step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention.

[0090] Figure 4 This is a flowchart illustrating an embodiment of the face data merging step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention.

[0091] Figure 5 This is a flowchart illustrating an embodiment of the face data deletion step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention.

[0092] Figure 6 A functional block diagram of an embodiment of the large-scale face retrieval dataset cleaning system for open sets provided by the present invention;

[0093] Figure 7 for Figure 6 A schematic diagram of the functional modules of an embodiment of the face similarity search module shown in the figure;

[0094] Figure 8 for Figure 6 A functional module diagram of one embodiment of the face data merging module shown in the figure;

[0095] Figure 9 for Figure 6 The diagram shows a functional module schematic of one embodiment of the face data deletion module.

[0096] Explanation of icon numbers:

[0097] 10. Face preprocessing module; 20. Face detection and key point detection module; 30. Face classification module; 40. Face feature extraction module; 50. Face similarity search module; 60. Face data merging module; 70. Face data deletion module; 51. First initialization unit; 52. First update unit; 53. Comparison unit; 54. First acquisition unit; 61. First extraction unit; 62. Concatenation unit; 63. Second initialization unit; 64. Second update unit; 65. Second acquisition unit; 66. Merging unit; 67. Third acquisition unit; 71. Second extraction unit; 72. Fourth acquisition unit; 73. Third initialization unit; 74. Third update unit; 75. Fifth acquisition unit; 76. Sixth acquisition unit; 77. Seventh acquisition unit. Detailed Implementation

[0098] To better understand the above technical solutions, the following will provide a detailed explanation of the technical solutions in conjunction with the accompanying drawings and specific implementation methods.

[0099] like Figure 1 As shown, the first embodiment of the present invention proposes a method for cleaning large-scale face retrieval datasets for open sets, including the following steps:

[0100] Step S100: Face preprocessing step. Use image processing tools to stitch together all images containing faces according to a canvas of a specified size to obtain stitched face images.

[0101] Step S200, face detection and key point detection step: perform face detection and key point detection on each stitched face image to obtain several face images and key points; use each key point to perform affine transformation on the corresponding face image to obtain each aligned face image.

[0102] Step S300: Face classification step, classify each aligned face image into usable faces and unusable faces, and remove unusable faces.

[0103] Face classification is used in the first round of image cleaning to classify each aligned face image into usable and unusable faces, and then removes the unusable faces. Usable faces are defined as frontal, clear, and recognizable. Unusable faces are defined as large profiles, blurry, and unrecognizable.

[0104] Step S400: Face feature extraction step. Face features are extracted from each available face to obtain face feature vectors. Combined with data analysis tools, each face feature vector is saved to the disk.

[0105] Step S500, the face similarity search step, is completed in parallel by several face similarity sub-searches. At the same time, the face feature vector is divided into several corresponding parts. Each of the several face similarity sub-searches is responsible for the similarity search task of one face feature vector and obtains a different identity ID folder. The different identity ID folders obtained by the several face similarity sub-searches are placed in the same folder to obtain the initial face folder.

[0106] Face similarity search is performed in parallel by several face similarity sub-searches, such as S1, S2, ..., Sn. Simultaneously, the face feature vector is divided into several corresponding parts, such as part 1, part 2, ..., part n. Each of the several face similarity sub-searches is responsible for the similarity search task of one face feature vector, and each obtains a different identity ID folder. The different identity ID folders obtained from the several face similarity sub-searches are placed in a single folder to obtain the initial face folder, as detailed in [link to details]. Figure 2 The number of paths is determined by hardware resources, such as CPU, GPU, and memory.

[0107] Step S600, Face Data Merging Step: Randomly select several face images from each subfolder of the initial face folder; extract face features from all randomly selected face images to obtain face feature vectors; concatenate the face feature vectors of the same identity ID to obtain concatenated face feature vectors; save all concatenated face feature vectors to disk using data analysis tools; calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the merging indexer; merge folders with similar distances less than the merging threshold to obtain a merged face temporary folder; update the merging threshold until the number of merged face temporary folders does not decrease significantly or the specified number of merging rounds is reached, thus obtaining a merged face folder.

[0108] Step S700, Face Data Deletion Step: Randomly select several face images from each subfolder of the merged face folder; extract face features from all randomly selected face images to obtain face feature vectors; concatenate face feature vectors with the same identity ID to obtain concatenated face feature vectors; save all concatenated face feature vectors to disk using data analysis tools; insert all concatenated face feature vectors into the deletion indexer and update the deletion indexer; calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the deletion indexer to obtain the minimum similarity distance and corresponding index number, and then store the minimum similarity distance and corresponding index number in the index. The number and corresponding file path are saved to the file to be deleted; based on the file to be deleted, all folders with fewer than the deletion threshold and not the most images in the folder are deleted to obtain a temporary folder of deleted faces, and the deletion threshold is updated according to a certain decay rate until the number of temporary folders of deleted faces basically does not decrease or stops after a specified number of rounds, to obtain a folder of deleted faces; the network parameters for face feature extraction are trained using the folder of deleted faces to obtain a better face feature extraction model, and the face feature extraction is further updated until the face feature extraction model reaches the expected accuracy or stops after a specified number of rounds, and the folder of deleted faces in the last round is the final cleaned face folder.

[0109] Further, please see Figure 2 and Figure 3 , Figure 2 This is a flowchart illustrating an embodiment of the face similarity search step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention. In this embodiment, step S500 includes:

[0110] Step S510: Initialize the search indexer.

[0111] The search indexer can be initialized using cosine distance index, inverted index, product quantization index, and inverted product quantization index, etc.

[0112] Step S520: Insert the first face feature vector into the search indexer, update the search indexer, and save the available faces corresponding to the first face feature vector to the specified folder.

[0113] Step S530: Calculate the similarity distance between each face feature vector and all face feature vectors in the search indexer to obtain the minimum similarity distance and the corresponding index number. If the minimum similarity distance is less than the set micro threshold, no operation is performed. If the minimum similarity distance is greater than the set micro threshold but less than the set small threshold, the available faces corresponding to the face feature vectors are saved to the folder with the same identity ID according to the index number. If the minimum similarity distance is greater than the small threshold, there are no face feature vectors with the same identity ID as the face feature vectors in the search indexer. Insert the face feature vectors into the search indexer, update the search indexer, and save the available faces corresponding to the face feature vectors to another specified folder.

[0114] Micro-thresholds can filter out nearly identical usable faces that need to be removed. Small thresholds can filter out usable faces that do not require cleaning.

[0115] Step S540: Obtain different ID folders for saving face images grouped by ID.

[0116] Preferably, see Figure 4 , Figure 4 This is a flowchart illustrating an embodiment of the face data merging step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention. In this embodiment, step S600 includes:

[0117] Step S610: Randomly select several face images from each subfolder of the initial face folder.

[0118] Step S620: Extract facial features from all randomly selected face images to obtain facial feature vectors, and concatenate the facial feature vectors of people with the same identity ID to obtain concatenated facial feature vectors; then, using data analysis tools, save all concatenated facial feature vectors to the disk.

[0119] Step S630: Initialize the merge indexer.

[0120] The merge indexer can be initialized using cosine distance index, inverted index, product quantization index, and inverted product quantization index.

[0121] Step S640: Insert all the stitched face feature vectors into the merge indexer and update the merge indexer.

[0122] Step S650: Calculate the similarity distance between each stitched face feature vector and all stitched face feature vectors in the merging indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be merged.

[0123] Step S660: Based on the files to be merged, merge folders smaller than the merging threshold to obtain a temporary folder of merged faces, and update the merging threshold according to a certain decay rate.

[0124] The merge threshold allows different folders with the same identity ID to be merged. It decreases progressively with each merge round, following a certain decay rate to ensure accuracy. The merge threshold is calculated using the following formula:

[0125] Merging threshold = Initial threshold - Decay rate * Number of merging rounds (1)

[0126] In formula (1), the merging threshold is used to merge different folders with the same identity ID.

[0127] Step S670: Repeat all the above operations until the number of merged face temporary folders does not decrease significantly or the specified number of rounds is reached, and obtain the merged face folders.

[0128] Further, please see Figure 5 , Figure 5 This is a flowchart illustrating an embodiment of the face data deletion step in the large-scale face retrieval dataset cleaning method for open sets provided by the present invention. In this embodiment, step S700 includes:

[0129] Step S710: Randomly select several face images from each subfolder of the merged face folder.

[0130] Step S720: Extract facial features from all randomly selected face images to obtain facial feature vectors, and concatenate the facial feature vectors of people with the same identity ID to obtain concatenated facial feature vectors; then, using data analysis tools, save all concatenated facial feature vectors to the disk.

[0131] Step S730: Initialize the delete indexer.

[0132] The delete indexer can be initialized with cosine distance index, inverted index, product quantization index, and inverted product quantization index, etc.

[0133] Step S740: Insert all the stitched face feature vectors into the delete indexer and update the delete indexer.

[0134] Step S750: Calculate the similarity distance between each stitched face feature vector and all stitched face feature vectors in the deletion indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be deleted.

[0135] Step S760: Based on the files to be deleted, delete all folders that are less than the deletion threshold and do not contain the most images, obtain a temporary folder of deleted faces, and update the deletion threshold according to a certain decay rate.

[0136] The deletion threshold allows you to delete all folders containing images less than the threshold that do not have the most images. The deletion threshold decreases progressively with each deletion round, following a certain decay rate to ensure accuracy. The deletion threshold is calculated using the following formula:

[0137] Deletion threshold = Initial threshold - Decay rate * Number of deletion rounds (2)

[0138] In formula (2), the deletion threshold is used to delete all folders that are less than the deletion threshold and do not have the most images in the folder.

[0139] Step S770: Repeat all the above operations until the number of deleted face temporary folders does not decrease significantly or reaches the specified number of rounds, and obtain the deleted face folder.

[0140] This embodiment provides a method for cleaning large-scale face retrieval datasets for open sets. This method includes seven steps: face preprocessing, face detection and keypoint detection, face classification, face feature extraction, face similarity search, face data merging, and face data deletion. It employs a multi-threshold, multi-path parallel, and multi-round iterative approach based on similarity search to automatically, accurately, and efficiently complete the entire process of cleaning large-scale face data without manual intervention. The search indexer based on similarity search uses an incremental construction method, ensuring that the face feature vector of the same identity ID exists only once in the search indexer. This efficiently places face data with the same identity ID in the same folder and face data with different identity IDs in different folders, while simultaneously removing essentially identical or highly similar face data. The cleaning method is suitable for different data volumes, meeting the cleaning needs of millions, tens of millions, or even hundreds of millions of data points.

[0141] Please see Figure 6 , Figure 6This is a functional block diagram of an embodiment of the large-scale face retrieval dataset cleaning system for open sets provided by the present invention. In this embodiment, the large-scale face retrieval dataset cleaning system for open sets includes a face preprocessing module 10, a face detection and key point detection module 20, a face classification module 30, a face feature extraction module 40, a face similarity search module 50, a face data merging module 60, and a face data deletion module 70. The face preprocessing module 10 is used to stitch together all images containing faces according to a specified size canvas using image processing tools to obtain stitched face images. The face detection and key point detection module 20 is used to perform face detection and key point detection on each stitched face image. Keypoint detection obtains several face images and facial keypoints; affine transformations are performed on the corresponding face images using each facial keypoint to obtain aligned face images; a face classification module 30 classifies the aligned face images into usable and unusable faces, and removes unusable faces; a face feature extraction module 40 extracts face features from each usable face, obtains face feature vectors, and saves each face feature vector to disk using data analysis tools; a face similarity search module 50 performs parallel face similarity sub-searches, while dividing the face feature vectors into corresponding parts; each of the face similarity sub-searches is responsible for one part of the face feature vector. The similarity search task obtains folders with different identity IDs. These folders are then combined into a single folder to create an initial face folder. A face data merging module 60 randomly selects several face images from each subfolder of the initial face folder. Face features are extracted from all randomly selected face images to obtain face feature vectors. Face feature vectors with the same identity ID are concatenated to obtain concatenated face feature vectors. Using data analysis tools, all concatenated face feature vectors are saved to disk. The similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the merging indexer is calculated, and vectors smaller than a certain value are selected. The merge threshold is used to merge folders that are to be merged, resulting in a temporary folder of merged faces. The merge threshold is then updated until the number of temporary folders of merged faces does not decrease significantly or the merge process stops after a specified number of rounds. The merged face folder is then obtained. The face data deletion module 70 randomly selects several face images from each subfolder of the merged face folder. Face features are extracted from all randomly selected face images to obtain face feature vectors. Face feature vectors with the same ID are concatenated to obtain concatenated face feature vectors. Using data analysis tools, all concatenated face feature vectors are saved to disk. Finally, all concatenated face feature vectors are inserted into the deletion indexer, and the deletion indexer is updated.The similarity distance between each stitched face feature vector and all stitched face feature vectors in the deletion indexer is calculated to obtain the minimum similarity distance and corresponding index number. This minimum similarity distance, corresponding index number, and corresponding file path are saved to the file to be deleted. Based on the file to be deleted, all folders with fewer than the deletion threshold and not the largest number of images are deleted, resulting in a temporary folder of deleted faces. The deletion threshold is updated according to a certain decay rate until the number of temporary folders of deleted faces essentially stops decreasing or a specified number of rounds is reached, resulting in a final folder of deleted faces. The network parameters for face feature extraction are trained using this folder to obtain a better face feature extraction model. This model is further updated until the face feature extraction model reaches the desired accuracy or a specified number of rounds is reached. The final folder of deleted faces is the final cleaned face folder.

[0142] Further, please see Figure 7 , Figure 7 for Figure 6 The diagram illustrates the functional modules of an embodiment of the face similarity search module. In this embodiment, the face similarity search module 50 includes a first initialization unit 51, a first update unit 52, a comparison unit 53, and a first acquisition unit 54. The first initialization unit 51 initializes the search indexer; the first update unit 52 inserts a first face feature vector into the search indexer, updates the search indexer, and saves the available faces corresponding to the first face feature vector to a specified folder; the comparison unit 53 calculates the similarity distance between each face feature vector and all face feature vectors in the search indexer to obtain the minimum similarity distance. If the minimum similarity distance is less than a set micro threshold, no operation is performed; if the minimum similarity distance is greater than the set micro threshold but less than the set small threshold, the available faces corresponding to the face feature vectors are saved to the folder with the same identity ID according to the index number; if the minimum similarity distance is greater than the small threshold, there is no face feature vector with the same identity ID as the face feature vector in the search indexer, the face feature vector is inserted into the search indexer, the search indexer is updated, and the available faces corresponding to the face feature vectors are saved to another specified folder; the first acquisition unit 54 is used to obtain different identity ID folders for saving face images in groups according to identity ID.

[0143] Preferably, see Figure 8 , Figure 8 for Figure 6The diagram illustrates the functional modules of an embodiment of the face data merging module. In this embodiment, the face data merging module 60 includes a first extraction unit 61, a splicing unit 62, a second initialization unit 63, a second update unit 64, a second acquisition unit 65, a merging unit 66, and a third acquisition unit 67. The first extraction unit 61 is used to randomly extract several face images from various subfolders of the initial face folder; the splicing unit 62 is used to extract face features from all randomly extracted face images to obtain face feature vectors, and splice face feature vectors with the same identity ID to obtain spliced ​​face feature vectors; combined with data analysis tools, all spliced ​​face feature vectors are saved to the disk; the second initialization unit 63 is used to initialize the merging indexer. The second update unit 64 is used to insert all the spliced ​​face feature vectors into the merging indexer and update the merging indexer; the second acquisition unit 65 is used to calculate the similarity distance between each spliced ​​face feature vector and all the spliced ​​face feature vectors in the merging indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be merged; the merging unit 66 is used to merge folders smaller than the merging threshold according to the file to be merged, obtain a merged face temporary folder, and update the merging threshold according to a certain decay rate; the third acquisition unit 67 is used to repeat all the above operations until the number of merged face temporary folders does not decrease significantly or stops after a specified number of rounds, and obtain the merged face folder.

[0144] The merge threshold allows different folders with the same identity ID to be merged. It decreases progressively with each merge round, following a certain decay rate to ensure accuracy. The merge threshold is calculated using the following formula:

[0145] Merging threshold = Initial threshold - Decay rate * Number of merging rounds (3)

[0146] In formula (3), the merging threshold is used to merge different folders with the same identity ID.

[0147] Further, please see Figure 9 , Figure 9 for Figure 6The diagram illustrates the functional modules of an embodiment of the face data deletion module 70. The face data deletion module 70 includes a second extraction unit 71, a fourth acquisition unit 72, a third initialization unit 73, a third update unit 74, a fifth acquisition unit 75, a sixth acquisition unit 76, and a seventh acquisition unit 77. Specifically, the second extraction unit 71 randomly extracts several face images from each subfolder of the merged face folder; the fourth acquisition unit 72 extracts face features from all randomly extracted face images to obtain face feature vectors, and concatenates face feature vectors with the same identity ID to obtain concatenated face feature vectors; and, using data analysis tools, saves all concatenated face feature vectors to the disk; the third initialization unit 73 initializes the deletion indexer; and the third update unit 74... The fifth acquisition unit 75 is used to insert all the stitched face feature vectors into the deletion indexer and update the deletion indexer; the sixth acquisition unit 76 is used to calculate the similarity distance between each stitched face feature vector and all the stitched face feature vectors in the deletion indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, corresponding index numbers and corresponding file paths to the file to be deleted; the seventh acquisition unit 77 is used to delete all folders that are less than the deletion threshold and do not have the most images in the folder, according to the file to be deleted, to obtain the deleted face temporary folder, and update the deletion threshold according to a certain decay rate; the seventh acquisition unit 77 is used to repeat all the above operations until the number of deleted face temporary folders basically does not decrease or reaches the specified number of rounds, to obtain the deleted face folder.

[0148] The deletion threshold allows you to delete all folders containing images less than the threshold that do not have the most images. The deletion threshold decreases progressively with each deletion round, following a certain decay rate to ensure accuracy. The deletion threshold is calculated using the following formula:

[0149] Deletion threshold = Initial threshold - Decay rate * Number of deletion rounds (4)

[0150] In formula (4), the deletion threshold is used to delete all folders that are less than the deletion threshold and do not have the most images in the folder.

[0151] This embodiment provides a large-scale face retrieval dataset cleaning system for open sets. The system includes a face preprocessing module, a face detection and keypoint detection module, a face classification module, a face feature extraction module, a face similarity search module, a face data merging module, and a face data deletion module. It employs a multi-threshold, multi-path parallel, and multi-round iterative method based on similarity search to automatically, accurately, and efficiently complete the entire process of cleaning large-scale face data without manual intervention. The search indexer based on similarity search uses an incremental construction method, ensuring that the face feature vector of the same identity ID exists only once in the search indexer. This efficiently places face data with the same identity ID in the same folder and face data with different identity IDs in different folders, while simultaneously removing essentially identical or highly similar face data. The cleaning method is suitable for different data volumes, meeting the cleaning needs of millions, tens of millions, or even hundreds of millions of data points.

[0152] The following is an example of cleaning an open set of facial image data with a scale of 100 million and an unknown number of facial identity IDs. The specific steps are as follows:

[0153] Step 1: Using an image processing tool (such as OpenCV), stitch all images containing faces together on a canvas of a specified size (e.g., 640*640) to obtain a stitched face image. By utilizing hardware resources, parallel processing can be employed to accelerate the image stitching process.

[0154] Step 2: Train a face detection and keypoint detection network (e.g., Yolov5Face) using an open-source face detection dataset (e.g., WildFace). Perform face detection and keypoint detection on all stitched face images to obtain several face images and keypoints. With appropriate hardware resources, multiple parallel processing paths can be used to accelerate face detection and keypoint detection.

[0155] Step 3: Train a face classification network (such as ResNet50) to classify the above face images into usable faces and unusable faces, and remove the unusable faces. Usable faces are characterized by being frontal, clear, and recognizable. Unusable faces are characterized by being large profiles, blurry, and unrecognizable. By utilizing hardware resources, multiple parallel processing paths can be used to accelerate face classification and obtain usable faces.

[0156] Step 4: Train a face feature extraction network (such as Insightface) using an open-source face dataset (such as Glint360K) to obtain an initial face feature extraction model.

[0157] Step 5: Extract features using a facial feature extraction model to obtain a facial feature vector (e.g., 128-dimensional). Then, using data analysis tools (e.g., Pandas and Feather), save the facial feature vector to disk. Depending on hardware resources, parallel processing can be employed to accelerate the facial feature extraction process.

[0158] Step 6: Using a similarity search tool (such as Faiss), the facial feature vectors are grouped one by one according to a given similarity threshold, and the corresponding facial images are stored in folders. Each group is considered as the same facial identity ID. All grouped folders constitute the initial facial folder. Depending on hardware resources, multiple parallel processing paths can be used to accelerate the grouping of facial images by facial identity ID.

[0159] Step 7: Randomly select several (e.g., 3) face images from the individual face IDs in the initial face folder. Using a trained face feature extraction network (e.g., Invisightface; note that the network parameters were obtained in the first round from the open-source dataset in Step 4), obtain face feature vectors (e.g., 128-dimensional). Concatenate the face feature vectors from several face images to obtain a high-dimensional face feature vector (e.g., 3*128=384-dimensional). Combine this with data analysis tools (e.g., Pandas and Feather) to save the high-dimensional face feature vector to disk; considering hardware resources, parallel processing can be used to accelerate face feature extraction.

[0160] Step 8: Using a similarity search tool (such as Faiss), calculate the similarity distance between each of the above high-dimensional face feature vectors and all high-dimensional face feature vectors in the merging indexer. Merge the folders to be merged that are less than the merging threshold (such as 2.0) to obtain a temporary folder of merged faces, and update the merging threshold (such as a decay rate of 0.01).

[0161] Step 9: Repeat steps 6 and 7 above until the number of merged face temporary folders is basically no longer decreasing or the specified number of merging rounds is stopped, and obtain the merged face folders.

[0162] Step 10: Randomly select several (e.g., 3) face images from the individual face IDs in the merged face folder. Using a trained face feature extraction network (e.g., Invisightface; note that the network parameters were obtained in the first round from the open-source dataset in step 4), obtain a face feature vector (e.g., 128-dimensional). Concatenate the face feature vectors from the several face images to obtain a high-dimensional face feature vector (e.g., 3*128=384-dimensional). Combine this with data analysis tools (e.g., Pandas and Feather) to save the high-dimensional face feature vector to disk; considering hardware resources, parallel processing can be used to accelerate the face feature extraction process.

[0163] Step 11: Using a similarity search tool (such as Faiss), calculate the similarity distance between each of the above high-dimensional face feature vectors and all high-dimensional face feature vectors in the deletion indexer. Delete all other folders with a similarity distance less than the deletion threshold (such as 3.0) and not having the largest number of folders to be deleted, obtain the temporary folder of deleted faces, and update the deletion threshold (such as a decay rate of 0.01). For specific operation procedures, please refer to point 9 in the specific technical solution.

[0164] Step 12: Repeat steps 9 and 10 above until the number of deleted faces in the temporary folder stops decreasing or the specified number of deletion rounds is reached, and the deleted faces folder is obtained.

[0165] Step 13: Using the deleted face folder mentioned above and combining it with Asian faces from open-source face datasets (such as Glint360K), create a format that can be used for training (such as the .idx and .rec formats required by the MXnet framework), and train the face feature extraction network until the network converges to obtain a new face feature extraction model.

[0166] Step 14: Repeat steps 5 to 13 above until the face feature extraction model reaches the desired accuracy or stops after a specified number of rounds. The final deleted face folder is the final cleaned face folder.

[0167] Although preferred embodiments of the invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including both the preferred embodiments and all changes and modifications falling within the scope of the invention. Clearly, those skilled in the art can make various alterations and modifications to the invention without departing from its spirit and scope. Thus, if these modifications and modifications of the invention fall within the scope of the claims and their equivalents, the invention is also intended to include these modifications and modifications.

Claims

1. A method for cleaning large-scale face retrieval datasets for open sets, characterized in that, Includes the following steps: The face preprocessing step involves using image processing tools to stitch together all images containing faces onto a canvas of a specified size to obtain a stitched face image. The face detection and key point detection steps involve performing face detection and key point detection on each stitched face image to obtain several face images and key points; and then performing an affine transformation on the corresponding face image using each key point to obtain each aligned face image. The face classification step involves classifying each aligned face image into usable faces and unusable faces, and removing the unusable faces. The facial feature extraction step involves extracting facial features from each of the available faces to obtain facial feature vectors, and then using data analysis tools to save each of the facial feature vectors to a disk. The face similarity search step is completed in parallel by several face similarity sub-searches. At the same time, the face feature vector is divided into several corresponding parts. Each of the several face similarity sub-searches is responsible for the similarity search task of one face feature vector and obtains a different identity ID folder. The different identity ID folders obtained by the several face similarity sub-searches are placed in the same folder to obtain the initial face folder. The face data merging step involves randomly selecting several face images from each subfolder of the initial face folder; extracting face features from all randomly selected face images to obtain face feature vectors; concatenating face feature vectors with the same identity ID to obtain concatenated face feature vectors; using data analysis tools, saving all concatenated face feature vectors to disk; calculating the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the merging indexer; merging folders with similar distances less than the merging threshold to obtain a merged face temporary folder; updating the merging threshold until the number of merged face temporary folders does not decrease significantly or stops after a specified number of merging rounds, thus obtaining a merged face folder. The face data deletion step involves: randomly selecting several face images from each subfolder of the merged face folder; extracting face features from all randomly selected face images to obtain face feature vectors; concatenating the face feature vectors of the same identity ID to obtain concatenated face feature vectors; using data analysis tools, saving all concatenated face feature vectors to disk; inserting all concatenated face feature vectors into a deletion indexer and updating the deletion indexer; calculating the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the deletion indexer to obtain the minimum similarity distance and corresponding index number; and saving the minimum similarity distance, the corresponding index number, and the corresponding file path to the file to be deleted. Based on the files to be deleted, all folders with fewer than the deletion threshold and not the largest number of images are deleted to obtain a temporary folder of deleted faces. The deletion threshold is then updated according to a certain decay rate until the number of temporary folders of deleted faces does not decrease significantly or stops after a specified number of rounds, resulting in a folder of deleted faces. The network parameters for face feature extraction are trained using the folder of deleted faces to obtain a better face feature extraction model. The face feature extraction model is further updated until it reaches the desired accuracy or stops after a specified number of rounds. The folder of deleted faces in the last round is the final cleaned face folder. The face similarity search steps include: Initialize the search indexer; The first face feature vector is inserted into the search indexer, the search indexer is updated, and the available faces corresponding to the first face feature vector are saved to the specified folder; Each facial feature vector is sequentially compared with all facial feature vectors in the search indexer to calculate the similarity distance, obtaining the minimum similarity distance and the corresponding index number. If the minimum similarity distance is less than a set micro threshold, no operation is performed. If the minimum similarity distance is greater than the set micro threshold but less than a set small threshold, the available face corresponding to the facial feature vector is saved to the folder with the same identity ID according to the index number. If the minimum similarity distance is greater than the small threshold, there is no facial feature vector with the same identity ID as the facial feature vector in the search indexer. The facial feature vector is then inserted into the search indexer, the search indexer is updated, and the available face corresponding to the facial feature vector is saved to another specified folder. Obtain the different identity ID folders that save face images grouped by identity ID.

2. The method for cleaning large-scale face retrieval datasets for open sets as described in claim 1, characterized in that, The face data merging step includes: Randomly select several face images from each subfolder of the initial face folder; Facial features are extracted from all randomly selected facial images to obtain facial feature vectors. The facial feature vectors of the same identity ID are then concatenated to obtain concatenated facial feature vectors. Using data analysis tools, all concatenated facial feature vectors are saved to disk. Initialize the merge indexer; Insert all the stitched face feature vectors into the merge indexer and update the merge indexer; The similarity distance between each of the spliced ​​face feature vectors and all the spliced ​​face feature vectors in the merging indexer is calculated to obtain the Top N closest similarity distances and corresponding index numbers. The similarity distances, the corresponding index numbers, and the corresponding file paths are then saved to the file to be merged. Based on the files to be merged, folders smaller than the merging threshold are merged to obtain a merged temporary face folder, and the merging threshold is updated according to a certain decay rate. Repeat all the above operations until the number of merged face temporary folders does not decrease significantly or reaches the specified number of rounds, and then obtain the merged face folders.

3. The method for cleaning large-scale face retrieval datasets for open sets as described in claim 2, characterized in that, The facial data deletion steps include: Randomly select several face images from each subfolder of the merged face folder; Facial features are extracted from all randomly selected facial images to obtain facial feature vectors. The facial feature vectors of the same identity ID are then concatenated to obtain concatenated facial feature vectors. Using data analysis tools, all concatenated facial feature vectors are saved to disk. Initialize the delete indexer; Insert all the stitched face feature vectors into the deletion indexer and update the deletion indexer; The similarity distance between each of the spliced ​​face feature vectors and all the spliced ​​face feature vectors in the deletion indexer is calculated to obtain the Top N closest similarity distances and corresponding index numbers, and the similarity distances, the corresponding index numbers and the corresponding file paths are saved to the file to be deleted; Based on the files to be deleted, all folders with fewer than the deletion threshold and not having the most images are deleted to obtain a temporary folder of deleted faces, and the deletion threshold is updated according to a certain decay rate. Repeat all the above operations until the number of deleted face temporary folders does not decrease significantly or reaches the specified number of rounds, and then obtain the deleted face folders.

4. The method for cleaning large-scale face retrieval datasets for open sets as described in claim 3, characterized in that, The merging threshold is calculated using the following formula: Merging threshold = Initial threshold - Decay rate * Number of merging rounds The merging threshold is used to merge different folders with the same identity ID; The deletion threshold is calculated using the following formula: Deletion threshold = Initial threshold - Decay rate * Number of deletion rounds The deletion threshold is used to delete all folders that have images less than the deletion threshold and do not contain the most images.

5. A system for cleaning large-scale face retrieval datasets for open sets, characterized in that, include: The face preprocessing module (10) is used to stitch together all images containing faces according to a specified size canvas using image processing tools to obtain a stitched face image. The face detection and key point detection module (20) is used to perform face detection and key point detection on each stitched face image to obtain several face images and key points; and to perform affine transformation on the corresponding face image using each key point to obtain each aligned face image. The face classification module (30) is used to classify each of the aligned face images into usable faces and unusable faces, and remove the unusable faces; The face feature extraction module (40) is used to extract face features from each of the available faces, obtain face feature vectors, and save each face feature vector to the disk in conjunction with data analysis tools; The face similarity search module (50) is used to complete the task in parallel by several face similarity sub-searches. At the same time, the face feature vector is divided into several corresponding parts. Each of the several face similarity sub-searches is responsible for the similarity search task of one face feature vector and obtains a different identity ID folder. The different identity ID folders obtained by the several face similarity sub-searches are placed in the same folder to obtain the initial face folder. The steps for face similarity search include: Initialize the search indexer; The first face feature vector is inserted into the search indexer, the search indexer is updated, and the available faces corresponding to the first face feature vector are saved to the specified folder; Each facial feature vector is sequentially compared with all facial feature vectors in the search indexer to calculate the similarity distance, obtaining the minimum similarity distance and the corresponding index number. If the minimum similarity distance is less than a set micro threshold, no operation is performed. If the minimum similarity distance is greater than the set micro threshold but less than a set small threshold, the available face corresponding to the facial feature vector is saved to the folder with the same identity ID according to the index number. If the minimum similarity distance is greater than the small threshold, there is no facial feature vector with the same identity ID as the facial feature vector in the search indexer. The facial feature vector is then inserted into the search indexer, the search indexer is updated, and the available face corresponding to the facial feature vector is saved to another specified folder. Obtain the different identity ID folders where face images are grouped and stored according to identity ID; The face data merging module (60) is used to randomly extract several face images from each subfolder of the initial face folder; extract face features from all randomly extracted face images to obtain face feature vectors; and concatenate the face feature vectors of the same identity ID to obtain concatenated face feature vectors; save all concatenated face feature vectors to the disk using data analysis tools; calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the merging indexer; merge folders that are less than the merging threshold to obtain a merged face temporary folder; and update the merging threshold until the number of merged face temporary folders does not decrease significantly or the specified number of merging rounds stops, thus obtaining a merged face folder. The face data deletion module (70) is used to randomly select several face images from each subfolder of the merged face folder; extract face features from all randomly selected face images to obtain face feature vectors, and concatenate the face feature vectors of the same identity ID to obtain concatenated face feature vectors; save all concatenated face feature vectors to the disk using data analysis tools; insert all concatenated face feature vectors into the deletion indexer and update the deletion indexer; calculate the similarity distance between each concatenated face feature vector and all concatenated face feature vectors in the deletion indexer to obtain the minimum similarity distance and the corresponding index number, and then store the minimum similarity distance and the corresponding index number in the data. The corresponding file paths are saved to the file to be deleted. Based on the file to be deleted, all folders with fewer than the deletion threshold and not the most images in the folder are deleted to obtain a temporary folder of deleted faces. The deletion threshold is updated according to a certain decay rate until the number of temporary folders of deleted faces basically does not decrease or stops after a specified number of rounds, thus obtaining a folder of deleted faces. The network parameters for face feature extraction are trained using the folder of deleted faces to obtain a better face feature extraction model. The face feature extraction is further updated until the face feature extraction model reaches the desired accuracy or stops after a specified number of rounds. The folder of deleted faces in the last round is the final cleaned face folder.

6. The large-scale face retrieval dataset cleaning system for open sets as described in claim 5, characterized in that, The face similarity search module (50) includes: The first initialization unit (51) is used to initialize the search indexer; The first update unit (52) is used to insert the first face feature vector into the search indexer, update the search indexer, and save the available face corresponding to the first face feature vector to the specified folder; The comparison unit (53) is used to calculate the similarity distance between each face feature vector and all face feature vectors in the search indexer in turn, to obtain the minimum similarity distance and the corresponding index number. If the minimum similarity distance is less than the set micro threshold, no operation is performed. If the minimum similarity distance is greater than the set micro threshold and less than the set small threshold, the available face corresponding to the face feature vector is saved to the folder with the same identity ID according to the index number. If the minimum similarity distance is greater than the small threshold, there is no face feature vector with the same identity ID as the face feature vector in the search indexer. The face feature vector is inserted into the search indexer, the search indexer is updated, and the available face corresponding to the face feature vector is saved to another specified folder. The first acquisition unit (54) is used to obtain the different identity ID folders that save face images in groups according to identity ID.

7. The large-scale face retrieval dataset cleaning system for open sets as described in claim 6, characterized in that, The face data merging module (60) includes: The first extraction unit (61) is used to randomly extract several face images from each subfolder of the initial face folder; The splicing unit (62) is used to extract facial features from all randomly selected face images to obtain facial feature vectors, and splice the facial feature vectors of the same identity ID to obtain spliced ​​facial feature vectors; combined with data analysis tools, all spliced ​​facial feature vectors are saved to the disk; The second initialization unit (63) is used to initialize the merge indexer; The second update unit (64) is used to insert all the spliced ​​face feature vectors into the merge indexer and update the merge indexer; The second acquisition unit (65) is used to calculate the similarity distance between each of the spliced ​​face feature vectors and all the spliced ​​face feature vectors in the merging indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, the corresponding index numbers and the corresponding file paths to the file to be merged. The merging unit (66) is used to merge folders smaller than the merging threshold according to the files to be merged, to obtain a merged temporary face folder, and to update the merging threshold according to a certain decay rate. The third acquisition unit (67) is used to repeat all the above operations until the number of merged face temporary folders does not decrease significantly or stops after a specified number of rounds, and obtains the merged face folders.

8. The large-scale face retrieval dataset cleaning system for open sets as described in claim 7, characterized in that, The face data deletion module (70) includes: The second extraction unit (71) is used to randomly extract several face images from each subfolder of the merged face folder; The fourth acquisition unit (72) is used to extract facial features from all randomly selected face images to obtain facial feature vectors, and to concatenate the facial feature vectors of the same identity ID to obtain concatenated facial feature vectors; and to save all concatenated facial feature vectors to the disk in conjunction with data analysis tools. The third initialization unit (73) is used to initialize the delete indexer; The third update unit (74) is used to insert all the spliced ​​face feature vectors into the deletion indexer and update the deletion indexer; The fifth acquisition unit (75) is used to calculate the similarity distance between each of the spliced ​​face feature vectors and all the spliced ​​face feature vectors in the deletion indexer, obtain the Top N closest similarity distances and corresponding index numbers, and save the similarity distances, the corresponding index numbers and the corresponding file paths to the file to be deleted; The sixth acquisition unit (76) is used to delete all folders that are less than the deletion threshold and do not have the most pictures in the folder, based on the file to be deleted, to obtain a temporary folder of deleted faces, and to update the deletion threshold according to a certain decay rate; The seventh acquisition unit (77) is used to repeat all the above operations until the number of the deleted face temporary folders does not decrease significantly or stops after a specified number of rounds, and obtain the deleted face folders.

9. The large-scale face retrieval dataset cleaning system for open sets as described in claim 8, characterized in that, The merging threshold is calculated using the following formula: Merging threshold = Initial threshold - Decay rate * Number of merging rounds The merging threshold is used to merge different folders with the same identity ID; The deletion threshold is calculated using the following formula: Deletion threshold = Initial threshold - Decay rate * Number of deletion rounds The deletion threshold is used to delete all folders that have images less than the deletion threshold and do not contain the most images.

Images