Face enrollment algorithm evaluation method and quality modeling algorithm evaluation method

Abstract

The present disclosure provides a face profiling algorithm evaluation method and a quality modeling algorithm evaluation method. The method comprises: profiling face image samples in an original data set by using a face profiling algorithm to obtain a sample identifier, a cluster, and a cluster master identifier corresponding to each face image sample; based on the sample identifier, the cluster, the cluster master identifier, and pre-defined cluster types and identifier types, the prediction results of the face profiling algorithm are counted to obtain a predetermined identifier number and cluster number for generating an evaluation instruction; based on the identifier number and the cluster number, a preset evaluation index is calculated, and the face profiling algorithm is evaluated according to the calculation result of the evaluation index. The present disclosure can comprehensively evaluate the face profiling algorithm through various evaluation indexes based on the identifier number and the cluster number, accurately evaluate the performance of the face profiling algorithm, and improve the evaluation effect of the face profiling algorithm.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for evaluating face archiving algorithms and a method for evaluating quality modeling algorithms. Background Technology

[0002] With the development of artificial intelligence and machine learning, facial recognition technology has been widely used in various fields. For example, in facial image archiving, facial archiving algorithms are used to classify facial images of the same person to form archives. Since the accuracy and recall rate of facial archiving algorithms significantly affect the effectiveness of facial image archiving, it is necessary to evaluate the performance of facial archiving algorithms in order to select those with better evaluation metrics for facial image archiving.

[0003] Currently, traditional evaluation schemes for face profiling algorithms typically only consider profiling metrics under a given fully labeled dataset, failing to evaluate weakly labeled datasets in real-world scenarios. Furthermore, traditional evaluation schemes use relatively singular evaluation metrics, usually only considering the difference between the number of profilings and the actual number of profilings, unable to comprehensively assess profiling purity, splitting degree, and other factors. Consequently, existing face profiling algorithm evaluation schemes cannot accurately evaluate the performance of face profiling algorithms, resulting in poor evaluation results. Summary of the Invention

[0004] In view of this, the present disclosure provides a face archiving algorithm evaluation method and a quality modeling algorithm evaluation method to solve the problems of existing technologies having a single evaluation index, being unable to comprehensively evaluate the performance of face archiving algorithms, and having relatively poor evaluation results.

[0005] A first aspect of this disclosure provides a method for evaluating a face archiving algorithm, comprising: archiving face image samples in an original dataset using a face archiving algorithm to obtain a sample identifier, cluster, and main identifier of each face image sample; statistically analyzing the prediction results of the face archiving algorithm based on the sample identifier, cluster, main identifier of the cluster, and predefined cluster type and identifier type to obtain a predetermined number of identifiers and clusters for generating evaluation instructions; calculating a preset evaluation index based on the number of identifiers and clusters; and evaluating the face archiving algorithm based on the calculation results of the evaluation index.

[0006] A second aspect of this disclosure provides a method for evaluating a quality modeling algorithm, comprising: archiving face image samples in an original dataset using a face archiving algorithm, and calculating the evaluation index corresponding to the face archiving algorithm to obtain a first evaluation index calculation result; sorting each face image sample in the original dataset using a quality modeling algorithm, and extracting a portion of face image samples from the original dataset to form a new dataset based on the sorting result; archiving face image samples in the new dataset using a face archiving algorithm, and calculating the evaluation index corresponding to the face archiving algorithm to obtain a second evaluation index calculation result; evaluating the quality modeling algorithm based on the first evaluation index calculation result and the second evaluation index calculation result; wherein, calculating the evaluation index corresponding to the face archiving algorithm includes: statistically analyzing the prediction results of the face archiving algorithm based on the sample identifier, cluster, and main identifier of each face image sample obtained after archiving, combined with a predefined cluster type and identifier type, to obtain a predetermined number of identifiers and clusters for generating evaluation instructions; and calculating the evaluation index based on the number of identifiers and clusters.

[0007] A third aspect of this disclosure provides a face archiving algorithm evaluation device, comprising: an archiving module configured to archive face image samples in an original dataset using a face archiving algorithm, obtaining a sample identifier, cluster, and main identifier of each face image sample; a statistics module configured to statistically analyze the prediction results of the face archiving algorithm based on the sample identifier, cluster, main identifier of the cluster, and predefined cluster type and identifier type, obtaining a predetermined number of identifiers and clusters for generating evaluation instructions; and an evaluation module configured to calculate preset evaluation indicators based on the number of identifiers and clusters, and evaluate the face archiving algorithm based on the calculation results of the evaluation indicators.

[0008] A fourth aspect of this disclosure provides a quality modeling algorithm evaluation apparatus, comprising: a first calculation module configured to archive face image samples in an original dataset using a face archiving algorithm, and to calculate an evaluation index corresponding to the face archiving algorithm to obtain a first evaluation index calculation result; a sorting module configured to sort each face image sample in the original dataset using a quality modeling algorithm, and to extract a portion of face image samples from the original dataset to form a new dataset based on the sorting result; and a second calculation module configured to use a face archiving algorithm to process the face image samples in the new dataset. The system establishes a profile and calculates the evaluation metrics corresponding to the face profile algorithm to obtain the calculation results of the second evaluation metrics. The evaluation module is configured to evaluate the quality modeling algorithm based on the calculation results of the first and second evaluation metrics. The first and second calculation modules are also used to statistically analyze the prediction results of the face profile algorithm based on the sample identifier, cluster, and main identifier of each face image sample obtained after profile establishment, combined with the predefined cluster type and identifier type, to obtain the predetermined number of identifiers and clusters used to generate evaluation instructions. The evaluation metrics are then calculated based on the number of identifiers and clusters.

[0009] The above-described at least one technical solution adopted in the embodiments of this disclosure can achieve the following beneficial effects:

[0010] By using a face profiling algorithm to archive face image samples in the original dataset, the sample identifier, cluster, and master identifier of each face image sample are obtained. Based on the sample identifier, cluster, master identifier, and predefined cluster and identifier types, the prediction results of the face profiling algorithm are statistically analyzed to obtain a predetermined number of identifiers and clusters for generating evaluation instructions. Based on the number of identifiers and clusters, preset evaluation indicators are calculated, and the face profiling algorithm is evaluated based on the calculation results of the evaluation indicators. This disclosure can comprehensively evaluate the face profiling algorithm through multiple evaluation indicators based on the number of identifiers and clusters, accurately evaluate the performance of the face profiling algorithm, and has better evaluation results. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 This is a flowchart illustrating the face profile evaluation algorithm provided in Embodiment 1 of this disclosure;

[0013] Figure 2This is a flowchart illustrating the quality modeling algorithm evaluation method provided in Embodiment 2 of this disclosure;

[0014] Figure 3 This is a schematic diagram of the structure of a face archiving algorithm evaluation device corresponding to Embodiment 1 of this disclosure;

[0015] Figure 4 This is a schematic diagram of the structure of a quality modeling algorithm evaluation device corresponding to Embodiment 2 of this disclosure;

[0016] Figure 5 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this disclosure. Detailed Implementation

[0017] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0018] As described in the background section, traditional face profiling and evaluation schemes typically only consider profiling metrics under a given fully labeled dataset, failing to evaluate weakly labeled datasets in real-world scenarios (e.g., samples without specific IDs, but known not to belong to a particular ID). Furthermore, traditional evaluation schemes use relatively singular evaluation metrics, usually only considering the difference between the number of profilings and the actual number of profilings, unable to comprehensively evaluate profiling purity, splitting degree, etc. Additionally, introducing quality modeling can improve profiling accuracy, a aspect that traditional evaluation schemes neglect.

[0019] In view of the problems in the prior art, the embodiments of this disclosure, in the face profiling evaluation task, not only consider the accuracy of profiling in real-world scenarios containing weakly labeled data, including purity and splitting, but also consider the evaluation of quality modeling algorithms. Therefore, this disclosure proposes evaluation schemes for two different algorithms. Embodiment one focuses on the evaluation of face profiling algorithms, aiming to achieve a comprehensive evaluation of the performance of face profiling algorithms by calculating evaluation indicators corresponding to the prediction results of the face profiling algorithms. Embodiment two addresses how to evaluate the quality modeling algorithm based on the evaluation indicators of the face profiling algorithm when using a quality modeling algorithm to rank face images by quality score in the face profiling evaluation task. The details of these two technical solutions are described below with reference to the accompanying drawings and specific embodiments.

[0020] Example 1

[0021] Figure 1This is a flowchart illustrating the face profile evaluation algorithm provided in Embodiment 1 of this disclosure. Figure 1 The facial recognition algorithm evaluation method can be executed by the server. For example... Figure 1 As shown, the evaluation method for this face profiling algorithm may specifically include:

[0022] S101, use the face archiving algorithm to archive the face image samples in the original dataset to obtain the sample identifier, cluster and main identifier of each face image sample;

[0023] S102, based on sample identifiers, clusters, cluster master identifiers, and predefined cluster types and identifier types, statistical analysis is performed on the prediction results of the face archiving algorithm to obtain the predetermined number of identifiers and clusters used to generate evaluation instructions;

[0024] S103, based on the number of identifiers and the number of clusters, calculates the preset evaluation indicators, and evaluates the face archiving algorithm based on the calculation results of the evaluation indicators.

[0025] Specifically, the face archiving algorithm evaluation method provided in Embodiment 1 of this disclosure is an evaluation of the face archiving algorithm used in a face archiving task. That is, after performing a face archiving task using one or more face archiving algorithms, the method provided in Embodiment 1 of this disclosure is used to calculate the index of the prediction result based on the prediction result of the face archiving algorithm, and the face archiving algorithm is evaluated based on the calculation result of the evaluation index. It should be noted that the face archiving algorithm evaluation method in Embodiment 1 of this disclosure is not limited to fixed face archiving scenarios and face archiving algorithms. Any scenario where a face archiving task is performed based on a face archiving algorithm is applicable to the technical solution of this disclosure. The face archiving algorithms of this disclosure include, but are not limited to, image grouping algorithms, portrait clustering algorithms, etc.

[0026] Furthermore, in this embodiment of the disclosure, the set of all images (face images) that are considered to be the same person in the prediction results is regarded as a cluster. A cluster can correspond to a file in the prediction results. A file can also be regarded as the set of all images belonging to the same face ID in all the ground truth labels (GT) in the original dataset. The main identifier (main ID) of the cluster refers to the ID in the GT corresponding to the sample with the most quantity in a cluster. This ID is used as the main ID of the cluster, that is, the cluster is predicted as this ID.

[0027] In some embodiments, a face archiving algorithm is used to archive face image samples in the original dataset, including: determining the face archiving algorithm to be evaluated, obtaining the original dataset in a real scene, using the face archiving algorithm to archive face image samples in the original dataset, obtaining a cluster corresponding to each face image sample, and establishing a face image archive based on the cluster; wherein, the original dataset contains face image samples with labels and / or face image samples without labels, and face image samples without labels are weakly labeled data.

[0028] Specifically, before evaluating the face profiling algorithm, the algorithm to be evaluated is first used to profil the pre-configured original dataset to obtain the prediction result corresponding to the algorithm. In practical applications, face profiling algorithms can employ face clustering algorithms, which group unlabeled face images. By comparing all faces within the set and analyzing their similarity values, people belonging to the same identity are grouped together.

[0029] Furthermore, this disclosure aims to evaluate the accuracy of face profiling algorithms in real-world scenarios, thus requiring consideration of data labeling conditions in these scenarios. In real-world datasets, face images often appear blurry, occluded, or in profile. Additionally, due to the complexity of the annotation process, incomplete labels may occur. Therefore, embodiments of this disclosure divide the samples (face images) in the original dataset into the following three scenarios:

[0030] In the first scenario, the face images in the dataset have clear ID labels. This scenario is the most common, and in general evaluation schemes, the face images in the dataset all have clear ID labels.

[0031] In the second type of dataset, the face images in this dataset do not have explicit ID labels (weakly labeled data), but it is known that the face image does not belong to a certain ID. This situation usually occurs because during the annotation process, only face images that do not belong to a certain ID in a certain cluster are removed, but the specific ID of the face image is not labeled.

[0032] In the third scenario, the dataset contains both face images with explicit ID labels and face images without explicit ID labels.

[0033] In some embodiments, the prediction results of the face archiving algorithm are statistically analyzed to obtain a predetermined number of identifiers and clusters for generating evaluation instructions. This includes: based on the prediction results corresponding to the face archiving algorithm, statistical analysis of the prediction results is performed using predefined cluster types and identifier types to obtain the number of identifiers and clusters for generating evaluation instructions. The number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of split identifiers, and the number of identifiers with missing data. The number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters.

[0034] Specifically, the predefined cluster types in this disclosure include predicted pure clusters, and the identifier types include predicted pure ID, predicted split ID, predicted missing ID, and predicted correct ID. The definitions of the cluster types and identifier types in this disclosure are described in detail below, and may include the following:

[0035] The definition of a predicted pure cluster is: if the proportion of samples belonging to the main ID in a cluster is greater than a certain proportion (e.g., 95%, where the proportion is an optional variable), then the cluster is defined as a predicted pure cluster.

[0036] The definition of a predicted pure ID is: if one or more clusters with this ID as the primary ID are pure clusters, then this ID is defined as a pure ID.

[0037] The predicted split ID is defined as follows: if more than one cluster uses this ID as the primary ID, then this ID is defined as a split ID;

[0038] The definition of a predicted missing ID is: if no cluster uses this ID as its primary ID, then this ID is defined as a missing ID.

[0039] A correctly predicted ID is defined as an ID that is a predicted pure ID and not a predicted split ID.

[0040] Furthermore, based on the prediction results of the face profiling algorithm, and according to the above definitions of cluster type and identifier type, the prediction results are statistically analyzed to obtain the number of identifiers and clusters used to generate evaluation instructions. In practical applications, the number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of split identifiers, and the number of identifiers with missing profiles. The number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters.

[0041] It should be noted that for samples in the dataset under the second scenario above, if the sample is registered in a cluster with an ID that does not belong as the primary ID, it is considered a misclassification; otherwise, the sample is ignored when performing statistical analysis on the prediction results.

[0042] In some embodiments, the evaluation metrics include filing accuracy, identifier purity, cluster purity, filing split rate, and filing recall. Based on the number of identifiers and the number of clusters, the preset evaluation metrics are calculated, including: using the ratio between the number of correctly predicted identifiers and the total number of identifiers as filing accuracy, the ratio between the number of predicted clean identifiers and the total number of identifiers as identifier purity, the ratio between the number of predicted clean clusters and the total number of predicted clusters as cluster purity, the ratio between the number of predicted split identifiers and the total number of identifiers as filing split rate, and the ratio between the number of predicted missing identifiers and the total number of identifiers as filing recall.

[0043] Specifically, based on the aforementioned embodiments, the number of identifiers and clusters obtained by statistically analyzing the prediction results are used to calculate each evaluation indicator using the number of identifiers and clusters, along with preset evaluation indicators, to obtain the value corresponding to each evaluation indicator. The calculation method of the evaluation indicators is described in detail below with reference to specific embodiments. This disclosure embodiment designs the following five evaluation indicators, and the specific calculation methods are as follows:

[0044]

[0045] Among them, the accuracy rate of record-keeping is the core indicator, which is used to comprehensively evaluate the accuracy of record-keeping.

[0046]

[0047] Among them, the ID purity index evaluates the purity of the prediction results from the perspective of ID in GT;

[0048]

[0049] Among them, the cluster purity index evaluates the purity of the prediction results from the perspective of the predicted clusters;

[0050]

[0051] Among them, the document split rate is used to assess and predict document splitting.

[0052]

[0053] The recall rate metric is used to assess the completeness of the predicted documentation.

[0054] Furthermore, after calculating the five evaluation metrics corresponding to the prediction results of each face archiving algorithm, the performance of the face archiving algorithm is evaluated according to the values ​​of the evaluation metrics, and finally the evaluation results corresponding to each face archiving algorithm are obtained.

[0055] According to the technical solution provided in Embodiment 1 of this disclosure, this disclosure constructs a complete evaluation strategy with multiple indicators by assigning weak labels to fuzzy labeled data in real-world scenarios. By defining five different evaluation indicators, it comprehensively evaluates the purity, splitting degree, and recall rate of face archiving algorithms. The evaluation scheme of this disclosure can consider the accuracy of archiving in real-world scenarios containing weakly labeled data, including purity, splitting degree, and recall rate. Therefore, it can accurately evaluate the performance of face archiving algorithms and improve the evaluation effect.

[0056] Example 2

[0057] Figure 2 This is a flowchart illustrating the quality modeling algorithm evaluation method provided in Embodiment 2 of this disclosure. Figure 2 The quality modeling algorithm evaluation method can be executed by the server. For example... Figure 2 As shown, the quality modeling algorithm evaluation method may specifically include:

[0058] S201, use the face archiving algorithm to archive the face image samples in the original dataset, and calculate the evaluation index corresponding to the face archiving algorithm to obtain the calculation result of the first evaluation index.

[0059] S202, use quality modeling algorithm to sort each face image sample in the original dataset, and extract a part of the face image samples from the original dataset to form a new dataset based on the sorting results;

[0060] S203, use the face archiving algorithm to archive the face image samples in the new dataset, and calculate the evaluation index corresponding to the face archiving algorithm to obtain the calculation result of the second evaluation index.

[0061] S204. The quality modeling algorithm is evaluated based on the calculation results of the first evaluation index and the second evaluation index.

[0062] The calculation of evaluation indicators for the face archiving algorithm includes: based on the sample identifier, cluster, and main identifier of each face image sample obtained after archiving, combined with the predefined cluster type and identifier type, the prediction results of the face archiving algorithm are statistically analyzed to obtain the predetermined number of identifiers and clusters used to generate evaluation instructions; and the evaluation indicators are calculated based on the number of identifiers and clusters.

[0063] Specifically, Embodiment 2 of this disclosure provides a method for evaluating the effectiveness of a quality modeling algorithm based on the calculation results of the five evaluation indicators when using a quality modeling algorithm to rank samples for quality scores in a face archiving task, based on the calculation method of the five evaluation indicators in Embodiment 1.

[0064] It should be noted that in Example 2, the same filing and evaluation scheme as provided in Example 1 was used when filing the original dataset and calculating the evaluation index for the new dataset. Therefore, the difference between Example 2 and Example 1 is that after filing the original dataset and calculating the first evaluation index, Example 2 uses a quality modeling algorithm to sort the samples in the original dataset, determines a new dataset based on the sorting results, files the new dataset and calculates the second evaluation index, and evaluates the performance of the quality modeling algorithm based on the changes in the evaluation index calculation results before and after.

[0065] In some embodiments, sorting each face image sample in the original dataset using a quality modeling algorithm includes: calculating a quality score for each face image sample in the original dataset using a quality modeling algorithm, and sorting the face image samples based on the quality score in descending order.

[0066] Specifically, the quality modeling algorithm of this disclosure is used to score each face image sample in the original dataset to obtain a quality score for each face image sample, and then sort the face image samples according to the quality scores. In practical applications, any quality modeling algorithm capable of scoring image samples is applicable to the technical solution of this disclosure.

[0067] In some embodiments, a portion of face image samples are extracted from the original dataset to form a new dataset based on the ranking results, including: removing face image samples with lower quality scores from the ranking results according to a preset ratio, and forming a new dataset with the remaining face image samples; or, adding face image samples with quality scores greater than a preset quality threshold into the new dataset according to a preset quality threshold.

[0068] Specifically, samples with lower quality scores (starting from the lowest quality score) are removed according to a certain proportion (e.g., 20%), and the remaining samples are used to form a new dataset. The face profiling algorithm is then used to archive the new dataset. Alternatively, in another embodiment, low-quality image samples can be removed according to a quality threshold, that is, only sample data with a quality threshold greater than a certain quality threshold are retained to form a new dataset.

[0069] In some embodiments, the quality modeling algorithm is evaluated based on the calculation results of the first evaluation index and the calculation results of the second evaluation index, including: determining the difference between the index in the calculation results of the second evaluation index and the index in the calculation results of the first evaluation index, using the difference as the index change, and using the index change to evaluate the quality modeling algorithm.

[0070] Specifically, the calculation results of the second evaluation index corresponding to the new dataset are compared with the calculation results of the first evaluation index corresponding to the original dataset. The quality modeling algorithm is evaluated by the degree of difference between the evaluation index before and after (the amount of index change). That is, by comparing the improvement of the evaluation index after archiving with the full amount of data (original dataset) and a part of high-quality data (new dataset), the performance of the quality modeling algorithm is evaluated. In practical applications, the greater the improvement of the evaluation index before and after, the better the effect of the quality modeling algorithm.

[0071] It should be emphasized that in Embodiment 2 of this application, the content that is the same as that in Embodiment 1 will not be repeated. It should be understood that the specific embodiments that are the same as those in Embodiment 1 mentioned above are also applicable to the technical solution corresponding to Embodiment 2. Therefore, the content that is the same as that in Embodiment 1 is also included in the technical solution of Embodiment 2.

[0072] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0073] Figure 3 This is a schematic diagram of the structure of a face archiving algorithm evaluation device corresponding to Embodiment 1 of this disclosure.

[0074] like Figure 3 As shown, the facial recognition algorithm evaluation device includes:

[0075] The archiving module 301 is configured to use a face archiving algorithm to archive face image samples in the original dataset, and obtain the sample identifier, cluster and main identifier of each face image sample.

[0076] The statistics module 302 is configured to perform statistics on the prediction results of the face archiving algorithm based on the sample identifier, cluster, main identifier of the cluster, and predefined cluster type and identifier type, to obtain a predetermined number of identifiers and clusters for generating evaluation instructions;

[0077] The evaluation module 303 is configured to calculate preset evaluation indicators based on the number of identifiers and the number of clusters, and evaluate the face archiving algorithm based on the calculation results of the evaluation indicators.

[0078] In some embodiments, Figure 3The filing module 301 determines the face filing algorithm to be evaluated and obtains the original dataset in the real scene. It uses the face filing algorithm to file the face image samples in the original dataset, obtains the cluster corresponding to each face image sample, and establishes a face image archive based on the cluster. The original dataset contains face image samples with labels and / or face image samples without labels. Face image samples without labels are weakly labeled data.

[0079] In some embodiments, Figure 3 The statistics module 302 uses the prediction results corresponding to the face archiving algorithm to perform statistics on the prediction results using predefined cluster types and identifier types to obtain the number of identifiers and clusters used to generate evaluation instructions. The number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of split identifiers, and the number of identifiers with missing data. The number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters.

[0080] In some embodiments, the evaluation metrics include documentation accuracy, identifier purity, cluster purity, documentation splitting rate, and documentation recall. Figure 3 The evaluation module 303 uses the ratio between the number of correctly predicted identifiers and the total number of identifiers as the filing accuracy, the ratio between the number of predicted clean identifiers and the total number of identifiers as the identifier purity, the ratio between the number of predicted clean clusters and the total number of predicted clusters as the cluster purity, the ratio between the number of predicted split identifiers and the total number of identifiers as the filing split rate, and the ratio between the number of predicted missing identifiers and the total number of identifiers as the filing recall rate.

[0081] Figure 4 This is a schematic diagram of the structure of the quality modeling algorithm evaluation device corresponding to Embodiment 2 of this disclosure.

[0082] like Figure 4 As shown, the quality modeling algorithm evaluation device includes:

[0083] The first calculation module 401 is configured to use a face archiving algorithm to archive face image samples in the original dataset, and to calculate the evaluation index corresponding to the face archiving algorithm to obtain the calculation result of the first evaluation index.

[0084] The sorting module 402 is configured to sort each face image sample in the original dataset using a quality modeling algorithm, and extract a portion of face image samples from the original dataset to form a new dataset based on the sorting results.

[0085] The second calculation module 403 is configured to use a face archiving algorithm to archive face image samples in a new dataset, and to calculate the evaluation index corresponding to the face archiving algorithm to obtain the calculation result of the second evaluation index.

[0086] Evaluation module 404 is configured to evaluate the quality modeling algorithm based on the calculation results of the first evaluation index and the calculation results of the second evaluation index.

[0087] The first calculation module 401 and the second calculation module 403 are further used to statistically analyze the prediction results of the face archiving algorithm based on the sample identifier, cluster and main identifier of each face image sample obtained after archiving, combined with the predefined cluster type and identifier type, to obtain the predetermined number of identifiers and clusters for generating evaluation instructions; and to calculate the evaluation index based on the number of identifiers and clusters.

[0088] In some embodiments, Figure 4 The sorting module 402 uses a quality modeling algorithm to calculate a quality score for each face image sample in the original dataset, and sorts the face image samples based on the quality score in descending order.

[0089] In some embodiments, Figure 4 The sorting module 402 removes face image samples with lower quality scores from the sorting results according to a preset ratio, and forms a new dataset with the remaining face image samples; or, according to a preset quality threshold, puts face image samples with quality scores higher than the quality threshold into the new dataset.

[0090] In some embodiments, Figure 4 The evaluation module 404 determines the difference between the indicators in the calculation results of the second evaluation indicator and the indicators in the calculation results of the first evaluation indicator, uses the difference as the indicator change, and uses the indicator change to evaluate the quality modeling algorithm.

[0091] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0092] Figure 5 This is a schematic diagram of the structure of the electronic device 5 provided in an embodiment of this disclosure. Figure 5 As shown, the electronic device 5 of this embodiment includes: a processor 501, a memory 502, and a computer program 503 stored in the memory 502 and executable on the processor 501. When the processor 501 executes the computer program 503, it implements the steps in the various method embodiments described above. Alternatively, when the processor 501 executes the computer program 503, it implements the functions of each module / unit in the various device embodiments described above.

[0093] For example, computer program 503 may be divided into one or more modules / units, which are stored in memory 502 and executed by processor 501 to perform the present disclosure. The one or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of computer program 503 in electronic device 5.

[0094] Electronic device 5 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 5 may include, but is not limited to, processor 501 and memory 502. Those skilled in the art will understand that... Figure 5 This is merely an example of electronic device 5 and does not constitute a limitation on electronic device 5. It may include more or fewer components than shown, or combine certain components, or different components. For example, electronic device may also include input / output devices, network access devices, buses, etc.

[0095] Processor 501 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0096] The memory 502 can be an internal storage unit of the electronic device 5, such as a hard disk or RAM. The memory 502 can also be an external storage device of the electronic device 5, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card. Furthermore, the memory 502 can include both internal and external storage units of the electronic device 5. The memory 502 is used to store computer programs and other programs and data required by the electronic device. The memory 502 can also be used to temporarily store data that has been output or will be output.

[0097] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0098] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0099] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this disclosure.

[0100] In the embodiments provided in this disclosure, it should be understood that the disclosed apparatus / computer devices and methods can be implemented in other ways. For example, the apparatus / computer device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. Multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] Furthermore, the functional units in the various embodiments of this disclosure can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0104] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method for evaluating face profile creation algorithms, characterized in that, include: The face archiving algorithm is used to archive the face image samples in the original dataset, and the sample identifier, cluster and main identifier of each face image sample are obtained. Based on the sample identifier, cluster, main identifier of the cluster, and predefined cluster type and identifier type, the prediction results of the face archiving algorithm are statistically analyzed to obtain a predetermined number of identifiers and clusters for generating evaluation instructions; Based on the number of identifiers and the number of clusters, preset evaluation indicators are calculated, and the face archiving algorithm is evaluated based on the calculation results of the evaluation indicators. The step of statistically analyzing the prediction results of the face archiving algorithm to obtain a predetermined number of identifiers and clusters for generating evaluation instructions includes: Based on the prediction results corresponding to the face archiving algorithm, the prediction results are statistically analyzed using predefined cluster types and identifier types to obtain the number of identifiers and clusters used to generate evaluation instructions; The number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of predicted split identifiers, and the number of predicted missing identifiers; the number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters. The evaluation metrics include document establishment accuracy, identifier purity, cluster purity, document establishment split rate, and document establishment recall rate. The calculation of the preset evaluation metrics based on the number of identifiers and clusters includes: The ratio between the predicted correct identifiers and the total number of identifiers is used as the filing accuracy rate; the ratio between the predicted clean identifiers and the total number of identifiers is used as the identifier purity; the ratio between the predicted clean clusters and the total predicted clusters is used as the cluster purity; the ratio between the predicted split identifiers and the total number of identifiers is used as the filing split rate; and the ratio between the predicted missed identifiers and the total number of identifiers is used as the filing recall rate. The definition of a predicted pure cluster is: if the proportion of samples belonging to the main ID in a cluster is greater than a certain proportion, then the cluster is defined as a predicted pure cluster; where the main ID of the cluster refers to the ID in the ground truth label GT corresponding to the sample with the largest number in a cluster, and this ID is used as the main ID of the cluster. The definition of a predicted pure ID is: if one or more clusters with this ID as their primary ID are predicted pure clusters, then this ID is defined as a predicted pure ID; the definition of a predicted split ID is: if more than one cluster has this ID as their primary ID, then this ID is defined as a predicted split ID; the definition of a predicted missing ID is: if no cluster has this ID as their primary ID, then this ID is defined as a predicted missing ID; the definition of a correctly predicted ID is: this ID belongs to the predicted pure IDs and does not belong to the predicted split IDs.

2. The method according to claim 1, characterized in that, The process of creating archives for face image samples in the original dataset using a face archiving algorithm includes: The face archiving algorithm to be evaluated is determined, and the original dataset in the real scene is obtained. The face archiving algorithm is used to archive the face image samples in the original dataset to obtain the cluster corresponding to each face image sample, and a face image archive is established based on the cluster. The original dataset contains labeled face image samples and / or unlabeled face image samples, wherein the unlabeled face image samples are weakly labeled data.

3. A quality modeling algorithm evaluation method, characterized in that, include: The face image samples in the original dataset are archived using a face archiving algorithm, and the evaluation index corresponding to the face archiving algorithm is calculated to obtain the calculation result of the first evaluation index. The quality modeling algorithm is used to sort each face image sample in the original dataset, and a portion of face image samples are extracted from the original dataset based on the sorting results to form a new dataset. The face archiving algorithm is used to archive the face image samples in the new dataset, and the evaluation index corresponding to the face archiving algorithm is calculated to obtain the calculation result of the second evaluation index. The quality modeling algorithm is evaluated based on the calculation results of the first evaluation index and the second evaluation index. The calculation of the evaluation indicators corresponding to the face archiving algorithm includes: Based on the sample identifier, cluster, and main identifier of each face image sample obtained after archiving, and combined with the predefined cluster type and identifier type, the prediction results of the face archiving algorithm are statistically analyzed to obtain the predetermined number of identifiers and clusters used to generate evaluation instructions; based on the number of identifiers and clusters, the evaluation index is calculated. The step of statistically analyzing the prediction results of the face archiving algorithm to obtain a predetermined number of identifiers and clusters for generating evaluation instructions includes: Based on the prediction results corresponding to the face archiving algorithm, the prediction results are statistically analyzed using predefined cluster types and identifier types to obtain the number of identifiers and clusters used to generate evaluation instructions; The number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of predicted split identifiers, and the number of predicted missing identifiers; the number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters. The evaluation metrics include document establishment accuracy, identifier purity, cluster purity, document establishment split rate, and document establishment recall rate. The calculation of the preset evaluation metrics based on the number of identifiers and clusters includes: The ratio between the predicted correct identifiers and the total number of identifiers is used as the filing accuracy rate; the ratio between the predicted clean identifiers and the total number of identifiers is used as the identifier purity; the ratio between the predicted clean clusters and the total predicted clusters is used as the cluster purity; the ratio between the predicted split identifiers and the total number of identifiers is used as the filing split rate; and the ratio between the predicted missed identifiers and the total number of identifiers is used as the filing recall rate. The definition of a predicted pure cluster is: if the proportion of samples belonging to the main ID in a cluster is greater than a certain proportion, then the cluster is defined as a predicted pure cluster; where the main ID of the cluster refers to the ID in the ground truth label GT corresponding to the sample with the largest number in a cluster, and this ID is used as the main ID of the cluster. The definition of a predicted pure ID is: if one or more clusters with this ID as their primary ID are predicted pure clusters, then this ID is defined as a predicted pure ID; the definition of a predicted split ID is: if more than one cluster has this ID as their primary ID, then this ID is defined as a predicted split ID; the definition of a predicted missing ID is: if no cluster has this ID as their primary ID, then this ID is defined as a predicted missing ID; the definition of a correctly predicted ID is: this ID belongs to the predicted pure IDs and does not belong to the predicted split IDs.

4. The method according to claim 3, characterized in that, The process of sorting each face image sample in the original dataset using a quality modeling algorithm includes: The quality modeling algorithm is used to calculate a quality score for each face image sample in the original dataset, and the face image samples are sorted based on the quality score from high to low.

5. The method according to claim 3, characterized in that, The step of extracting a portion of face image samples from the original dataset to form a new dataset based on the sorting results includes: According to a preset ratio, remove the face image samples with lower quality scores from the sorting results, and combine the remaining face image samples into a new dataset; Alternatively, according to a preset quality threshold, face image samples with quality scores greater than the quality threshold can be added to the new dataset.

6. The method according to claim 3, characterized in that, The evaluation of the quality modeling algorithm based on the calculation results of the first evaluation index and the second evaluation index includes: The difference between the indicator in the calculation result of the second evaluation indicator and the indicator in the calculation result of the first evaluation indicator is determined, and the difference is used as the indicator change amount. The quality modeling algorithm is evaluated using the indicator change amount.

7. A facial recognition algorithm evaluation device, characterized in that, include: The archiving module is configured to use a face archiving algorithm to archive face image samples in the original dataset, and obtain the sample identifier, cluster, and main identifier of each face image sample. The statistics module is configured to perform statistics on the prediction results of the face archiving algorithm based on the sample identifier, cluster, main identifier of the cluster, and predefined cluster type and identifier type, to obtain a predetermined number of identifiers and clusters for generating evaluation instructions; The evaluation module is configured to calculate preset evaluation indicators based on the number of identifiers and the number of clusters, and evaluate the face archiving algorithm based on the calculation results of the evaluation indicators, wherein the evaluation indicators include archiving accuracy, identifier purity, cluster purity, archiving split rate and archiving recall rate. The statistics module is used to perform statistics on the prediction results based on the face archiving algorithm, using predefined cluster types and identifier types, to obtain the number of identifiers and clusters used to generate evaluation instructions. The number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of split identifiers, and the number of mispredicted identifiers. The number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters. The evaluation module is used to take the ratio between the number of correctly predicted identifiers and the total number of identifiers as the filing accuracy, the ratio between the number of predicted clean identifiers and the total number of identifiers as the identifier purity, the ratio between the number of predicted clean clusters and the total number of predicted clusters as the cluster purity, the ratio between the number of predicted split identifiers and the total number of identifiers as the filing split rate, and the ratio between the number of predicted missed identifiers and the total number of identifiers as the filing recall rate. The definition of a predicted pure cluster is: if the proportion of samples belonging to the main ID in a cluster is greater than a certain proportion, then the cluster is defined as a predicted pure cluster; where the main ID of the cluster refers to the ID in the ground truth label GT corresponding to the sample with the largest number in a cluster, and this ID is used as the main ID of the cluster. The definition of a predicted pure ID is: if one or more clusters with this ID as their primary ID are predicted pure clusters, then this ID is defined as a predicted pure ID; the definition of a predicted split ID is: if more than one cluster has this ID as their primary ID, then this ID is defined as a predicted split ID; the definition of a predicted missing ID is: if no cluster has this ID as their primary ID, then this ID is defined as a predicted missing ID; the definition of a correctly predicted ID is: this ID belongs to the predicted pure IDs and does not belong to the predicted split IDs.

8. A quality modeling algorithm evaluation device, characterized in that, include: The first calculation module is configured to archive face image samples in the original dataset using a face archiving algorithm, and to calculate the evaluation index corresponding to the face archiving algorithm to obtain the calculation result of the first evaluation index, wherein the evaluation index includes archiving accuracy, label purity, cluster purity, archiving split rate and archiving recall. The sorting module is configured to sort each face image sample in the original dataset using a quality modeling algorithm, and extract a portion of face image samples from the original dataset to form a new dataset based on the sorting results. The second calculation module is configured to use the face archiving algorithm to archive the face image samples in the new dataset, and to calculate the evaluation index corresponding to the face archiving algorithm to obtain the calculation result of the second evaluation index. The evaluation module is configured to evaluate the quality modeling algorithm based on the calculation results of the first evaluation index and the calculation results of the second evaluation index. The first calculation module and the second calculation module are further configured to statistically analyze the prediction results of the face archiving algorithm based on the sample identifier, cluster, and main identifier of each face image sample obtained after archiving, combined with predefined cluster types and identifier types, to obtain a predetermined number of identifiers and clusters for generating evaluation instructions; and to calculate the evaluation index based on the number of identifiers and clusters. The first calculation module and the second calculation module are used to perform statistical analysis on the prediction results corresponding to the face archiving algorithm using predefined cluster types and identifier types to obtain the number of identifiers and the number of clusters used to generate evaluation instructions. The number of identifiers includes the total number of identifiers corresponding to the original dataset, the number of correctly predicted identifiers, the number of correctly predicted identifiers, the number of split identifiers, and the number of mispredicted identifiers. The number of clusters includes the number of correctly predicted clusters and the total number of predicted clusters. The first calculation module and the second calculation module are used to take the ratio between the predicted correct identifiers and the total number of identifiers as the filing accuracy, the ratio between the predicted clean identifiers and the total number of identifiers as the identifier purity, the ratio between the predicted clean clusters and the total predicted clusters as the cluster purity, the ratio between the predicted split identifiers and the total number of identifiers as the filing split rate, and the ratio between the predicted missed identifiers and the total number of identifiers as the filing recall rate. The definition of a predicted pure cluster is: if the proportion of samples belonging to the main ID in a cluster is greater than a certain proportion, then the cluster is defined as a predicted pure cluster; where the main ID of the cluster refers to the ID in the ground truth label GT corresponding to the sample with the largest number in a cluster, and this ID is used as the main ID of the cluster. The definition of a predicted pure ID is: if one or more clusters with this ID as their primary ID are predicted pure clusters, then this ID is defined as a predicted pure ID; the definition of a predicted split ID is: if more than one cluster has this ID as their primary ID, then this ID is defined as a predicted split ID; the definition of a predicted missing ID is: if no cluster has this ID as their primary ID, then this ID is defined as a predicted missing ID; the definition of a correctly predicted ID is: this ID belongs to the predicted pure IDs and does not belong to the predicted split IDs.

Images