Methods and apparatus for evaluating the performance of face recognition models

Abstract

This disclosure relates to the field of face recognition technology, and provides a method and apparatus for evaluating the performance of a face recognition model. The method includes: acquiring a first image test set, a second image test set, and a third image test set; calculating a face comparison index of the face recognition model based on the first image test set; calculating a closed-set face retrieval index of the face recognition model based on the second image test set; calculating an open-set face retrieval index of the face recognition model based on the third image test set; and evaluating the performance of the face recognition model based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index. By employing the above technical means, the problem of the inability to accurately evaluate the performance of face recognition models in existing technologies is solved.

Description

Technical Field

[0001] This disclosure relates to the field of face recognition technology, and in particular to a method and apparatus for evaluating the performance of a face recognition model. Background Technology

[0002] Facial recognition models have a wide range of applications. To select the best facial recognition model from among many, or to evaluate a particular model, it is necessary to assess its performance. Currently, there is no proposed method for evaluating the performance of facial recognition models specifically for various scenarios. We can only use commonly used model performance metrics, leading to inaccurate evaluations. Common performance metrics include accuracy, precision, recall, and false positive rate.

[0003] In realizing the present invention, the inventors discovered at least the following technical problem in the related technology: the inability to accurately evaluate the performance of face recognition models. Summary of the Invention

[0004] In view of this, the present disclosure provides a method, apparatus, electronic device and computer-readable storage medium for evaluating the performance of a face recognition model, in order to solve the problem in the prior art that the performance of a face recognition model cannot be accurately evaluated.

[0005] A first aspect of this disclosure provides a method for evaluating the performance of a face recognition model, comprising: acquiring a first image test set, a second image test set, and a third image test set; calculating a face comparison index of the face recognition model based on the first image test set; calculating a closed-set face retrieval index of the face recognition model based on the second image test set; calculating an open-set face retrieval index of the face recognition model based on the third image test set; and evaluating the performance of the face recognition model based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index.

[0006] A second aspect of this disclosure provides an apparatus for evaluating the performance of a face recognition model, comprising: an acquisition module configured to acquire a first image test set, a second image test set, and a third image test set; a first calculation module configured to calculate a face comparison index of the face recognition model based on the first image test set; a second calculation module configured to calculate a closed-set face retrieval index of the face recognition model based on the second image test set; a third calculation module configured to calculate an open-set face retrieval index of the face recognition model based on the third image test set; and an evaluation module configured to evaluate the performance of the face recognition model based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index.

[0007] A third aspect of this disclosure provides an electronic device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method described above.

[0008] A fourth aspect of this disclosure provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the above-described method.

[0009] The beneficial effects of this disclosure embodiment compared with the prior art are as follows: because this disclosure embodiment obtains a first image test set, a second image test set, and a third image test set; calculates the face comparison index of the face recognition model based on the first image test set; calculates the closed-set face retrieval index of the face recognition model based on the second image test set; calculates the open-set face retrieval index of the face recognition model based on the third image test set; and evaluates the performance of the face recognition model based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index, the above technical means can solve the problem in the prior art that it is impossible to accurately evaluate the performance of the face recognition model, thereby improving the accuracy of evaluating the performance of the face recognition model. Attached Figure Description

[0010] To more clearly illustrate the technical solutions in the embodiments of this disclosure, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure;

[0012] Figure 2 This is a flowchart illustrating a method for evaluating the performance of a face recognition model provided in an embodiment of this disclosure;

[0013] Figure 3 This is a schematic diagram of the structure of a face recognition model performance evaluation device provided in an embodiment of this disclosure;

[0014] Figure 4 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0015] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, so as to provide a thorough understanding of the embodiments of this disclosure. However, those skilled in the art will understand that this disclosure may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this disclosure with unnecessary detail.

[0016] The following will describe in detail, with reference to the accompanying drawings, a method and apparatus for evaluating the performance of a face recognition model according to embodiments of the present disclosure.

[0017] Figure 1 This is a schematic diagram illustrating an application scenario of an embodiment of this disclosure. The application scenario may include terminal devices 101, 102, and 103, server 104, and network 105.

[0018] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices with displays that support communication with server 104, including but not limited to smartphones, tablets, laptops, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the aforementioned electronic devices. Terminal devices 101, 102, and 103 can be implemented as multiple software programs or software modules, or as a single software program or software module; this disclosure does not impose any limitations on this. Furthermore, various applications can be installed on terminal devices 101, 102, and 103, such as data processing applications, instant messaging tools, social platform software, search applications, shopping applications, etc.

[0019] Server 104 can be a server that provides various services, such as a backend server that receives requests sent by terminal devices with which it has established communication connections. This backend server can receive and analyze the requests sent by the terminal devices and generate processing results. Server 104 can be a single server, a server cluster consisting of several servers, or a cloud computing service center. This embodiment of the disclosure does not impose any limitations on these aspects.

[0020] It should be noted that server 104 can be either hardware or software. When server 104 is hardware, it can be various electronic devices that provide various services to terminal devices 101, 102, and 103. When server 104 is software, it can be multiple software programs or software modules that provide various services to terminal devices 101, 102, and 103, or it can be a single software program or software module that provides various services to terminal devices 101, 102, and 103. This disclosure does not limit the scope of the embodiments.

[0021] Network 105 can be a wired network using coaxial cable, twisted pair, and fiber optic connection, or it can be a wireless network that enables interconnection of various communication devices without wiring, such as Bluetooth, Near Field Communication (NFC), Infrared, etc. This disclosure does not limit the scope of the network.

[0022] Users can establish a communication connection with server 104 via network 105 through terminal devices 101, 102, and 103 to receive or send information, etc. It should be noted that the specific types, quantities, and combinations of terminal devices 101, 102, and 103, server 104, and network 105 can be adjusted according to the actual needs of the application scenario, and this disclosure embodiment does not impose any limitations on this.

[0023] Figure 2 This is a flowchart illustrating a method for evaluating the performance of a face recognition model provided in an embodiment of this disclosure. Figure 2 The performance evaluation method for face recognition models can be derived from... Figure 1 The computer or server, or the software on the computer or server, executes the command. For example... Figure 2 As shown, the performance evaluation method for this face recognition model includes:

[0024] S201, Obtain the first image test set, the second image test set, and the third image test set;

[0025] S202, Based on the first image test set, calculate the face comparison index of the face recognition model;

[0026] S203, Based on the second image test set, calculate the closed-set face retrieval index of the face recognition model;

[0027] S204, Calculate the open-set face retrieval index of the face recognition model based on the third image test set;

[0028] S205 evaluates the performance of the face recognition model based on face comparison metrics, closed-set face retrieval metrics, and open-set face retrieval metrics.

[0029] The first, second, and third image test sets each contain multiple identifiers and multiple images under each identifier. Each identifier in the first image test set has no background image, each identifier in the second image test set has one background image, and each identifier in the third image test set with a preset ratio has one background image, while other identifiers have no background images (identifiers with a ratio one minus the preset ratio in the third image test set have no background images).

[0030] An identifier represents a person, and the multiple images associated with that identifier are multiple images of that person. The base image is a special image; for example, in a work attendance system, when each employee registers at the attendance machine, the image collected by the machine is the employee's base image.

[0031] Each identifier in the second image test set has a background image, so the second image test set can also be called the closed set test set. The index calculated based on the second image test set is called the closed set face retrieval index. In the third image test set, some identifiers have a background image, while others do not. Therefore, the third image test set can also be called the open set test set. The index calculated based on the third image test set is called the open set face retrieval index.

[0032] The process of calculating the corresponding indicators based on the first image test set, the second image test set, and the third image test set can also be understood as generating three tasks: a task to calculate the face comparison indicator, a task to calculate the closed-set face retrieval indicator, and a task to calculate the open-set face retrieval indicator; and then executing the three tasks to obtain the face comparison indicator, the closed-set face retrieval indicator, and the open-set face retrieval indicator.

[0033] Obviously, the technical solution provided in this disclosure can be applied to any image processing technology field. In other words, the technical solution provided in this disclosure can evaluate the performance of any image recognition model. It only requires replacing the face comparison metrics, closed-set face retrieval metrics, and open-set face retrieval metrics with metrics for the corresponding image types.

[0034] According to the technical solution provided in this disclosure, a first image test set, a second image test set, and a third image test set are obtained; based on the first image test set, a face comparison index of the face recognition model is calculated; based on the second image test set, a closed-set face retrieval index of the face recognition model is calculated; based on the third image test set, an open-set face retrieval index of the face recognition model is calculated; and based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index, the performance of the face recognition model is evaluated. Therefore, by adopting the above technical means, the problem of not being able to accurately evaluate the performance of the face recognition model in the prior art can be solved, thereby improving the accuracy of evaluating the performance of the face recognition model.

[0035] Based on the first image test set, the face comparison index of the face recognition model is calculated, including: in the process of using the face recognition model to identify whether any two images in the first image test set belong to the same identifier, the number of false passes, the number of correct passes, the number of false rejections, the total number of any two images not belonging to the same identifier, and the total number of any two images belonging to the same identifier are counted; the false pass rate is calculated based on the number of false passes and the total number of any two images not belonging to the same identifier; the correct pass rate is calculated based on the number of correct passes and the total number of any two images belonging to the same identifier; the false rejection rate is calculated based on the number of false rejections and the total number of any two images belonging to the same identifier; and the face comparison index is calculated based on the false pass rate, the correct pass rate, and the false rejection rate.

[0036] In this embodiment of the disclosure, given any two images, the system identifies whether the two images belong to the same identifier, that is, to determine whether the two images depict the same person. The determination result includes the following four cases:

[0037] It's actually the same person. They are not actually the same person. Judged to be the same person Passed correctly Error passed It was determined that they were not the same person. Error rejection Correct Rejection

[0038] Throughout the entire discrimination process, the following statistics are recorded: the number of incorrectly passed images, the number of correctly passed images, the number of incorrectly rejected images, the total number of any two images that do not belong to the same identifier, and the total number of any two images that belong to the same identifier.

[0039] The false pass rate can be obtained by dividing the number of false passes by the total number of times any two images do not belong to the same identifier; the correct pass rate can be obtained by dividing the number of correct passes by the total number of times any two images belong to the same identifier; and the false rejection rate can be obtained by dividing the number of false rejections by the total number of times any two images belong to the same identifier.

[0040] The face comparison metrics are calculated based on the false pass rate, correct pass rate, and false rejection rate. This includes: determining a first threshold corresponding to a false pass rate of 1Ek; calculating a security index based on the first threshold and the correct pass rate, where 1Ek is scientific notation, meaning the decimal point after 1 is moved to the right by k places, and k is an integer greater than or equal to 2; determining a second threshold corresponding to a false rejection rate of 1Ek; and calculating a pass rate index based on the second threshold and the false pass rate. The face comparison metrics include both a security index and a pass rate index.

[0041] A first threshold is determined when the false pass rate is 1Ek. A security index is calculated based on the first threshold and the correct pass rate. This can be understood as evaluating the correct pass rate when the false pass rate is low, and the evaluation result is the security index. A second threshold is determined when the false rejection rate is 1Ek. A passability index is calculated based on the second threshold and the false pass rate. This can be understood as evaluating the false pass rate when the false rejection rate is low, and the evaluation result is the passability index.

[0042] The correct pass rate is denoted as TPR, the false pass rate as FPR, and the false rejection rate as FRR. The calculation process described above can be expressed as follows:

[0043] TPR@FPR=1Ek(k=2, 3, 4, 5, 6...)

[0044] FPR@FRR=1Ek(k=2, 3, 4, 5, 6...)

[0045] k can be set by yourself.

[0046] Based on the second image test set, the closed-set face retrieval index of the face recognition model is calculated, including: during the process of retrieving the background image corresponding to any image in the second image test set using the face recognition model, counting the number of images whose similarity to their corresponding background image ranks within the top N of all similarities, the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and whose similarity to their corresponding background image is greater than a third threshold, and the total number of pairs of images and background images; calculating the first pass index based on the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and the total number of pairs of images and background images; calculating the second pass index based on the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and whose similarity to their corresponding background image is greater than a third threshold, and the total number of pairs of images and background images; wherein, images and their background images under the same identifier have a corresponding relationship, the closed-set face retrieval index includes: the first pass index and the second pass index.

[0047] During the process of retrieving the background image corresponding to any image in the second image test set using a face recognition model, the face recognition model calculates a similarity score between the image and the background image each time it performs a retrieval (i.e., compares any image with any background image). N is a positive integer and can be set by the user. The total number of pairs of images and background images can be understood as the number of pairs that can be formed between any image and any background image.

[0048] The first pass criterion can be obtained by dividing the number of images whose similarity to their corresponding base image ranks within the top N of all similarity scores by the total number of pairs of images and base images. The second pass criterion can be obtained by dividing the number of images whose similarity to their corresponding base image ranks within the top N of all similarity scores and whose similarity to their corresponding base image is greater than the third threshold by the total number of pairs of images and base images.

[0049] Based on the third image test set, the open-set face retrieval index of the face recognition model is calculated, including: during the process of retrieving the background image corresponding to any image in the third image test set using the face recognition model, the number of false successes, the number of correct successes, the number of correct rejections, the number of identifiers with background images, and the number of identifiers without background images are counted; the false success rate is calculated based on the number of false successes and the number of identifiers without background images; the correct success rate is calculated based on the number of correct successes and the number of identifiers with background images; the recognition success rate is calculated based on the number of correct successes, the number of correct rejections, the number of identifiers with background images, and the number of identifiers without background images; and the open-set face retrieval index is calculated based on the false success rate, the correct success rate, and the recognition success rate.

[0050] When using a face recognition model to retrieve the background image corresponding to any image in the third image test set, the following four situations exist:

[0051] This person exists in the base map This person does not have a base map. This person was found Successful retrieval (passed correctly) Error retrieval (error passed) This person was not found. Error rejection Correct Rejection

[0052] The numbers of false passes, correct passes, and correct rejections are similar to the "false pass rate, correct pass rate, and false rejection rate" mentioned above. The concepts with the same names are similar, but the scenarios are different, so they will not be elaborated on here.

[0053] The false pass rate can be obtained by dividing the number of false passes by the number of identifiers without a base map; the correct pass rate can be obtained by dividing the number of correct passes by the number of identifiers with a base map; and the recognition success rate can be obtained by dividing the sum of the number of correct passes and the number of correct rejections by the sum of the number of identifiers with a base map and the number of identifiers without a base map.

[0054] Based on the false pass rate, correct pass rate, and recognition success rate, open-set face retrieval metrics are calculated, including: determining the first threshold corresponding to a false pass rate of 1Ek; calculating a security index based on the first threshold and the correct pass rate, where 1Ek is scientific notation, representing moving the decimal point after 1 to the right by k places, and k being an integer greater than or equal to 2; and calculating a success index based on the first threshold and the recognition success rate. The open-set face retrieval metrics include both security and success indices.

[0055] The success index is calculated based on the first threshold and the recognition success rate. This can be understood as evaluating the recognition success rate when the false pass rate is low, and the evaluation result is the success index.

[0056] The correct pass rate is denoted as TPR, the incorrect pass rate as FPR, and the success rate indicator as ACC. The above calculation process can be expressed as follows:

[0057] TPR@FPR=1Ek(k=2, 3, 4, 5, 6...)

[0058] ACC@FPR=1Ek(k=2, 3, 4, 5, 6...)

[0059] After obtaining the first, second, and third image test sets, the following processing can be performed on them: Image quality is estimated using common face quality assessment algorithms, and low-quality images are directly removed to improve the accuracy of comparison or retrieval. The effectiveness of image filtering is evaluated during this process: all images in each test set are ranked by quality, and after removing a certain proportion of low-quality images, the remaining images are evaluated for performance metrics. A greater improvement in metrics indicates better model performance.

[0060] The performance of a face recognition model can be evaluated based on face comparison metrics, closed-set face retrieval metrics, and open-set face retrieval metrics. This can be achieved by weighting and summing the face comparison metrics, closed-set face retrieval metrics, and open-set face retrieval metrics according to the target weights, and using the summation result as the performance metric of the face recognition model.

[0061] It should be noted that the target weights are different in different scenarios. For example, in the face comparison scenario, the face comparison index has the largest weight; in the attendance tracking scenario where security requirements are not high, the closed-set face retrieval index has the largest weight; and in the attendance tracking scenario where security requirements are high, the open-set face retrieval index has the largest weight (because in the attendance tracking scenario where security requirements are not high, almost everyone who is clocking in has already registered on the clock-in machine, and there are few people who are faking attendance, so the closed-set face retrieval index has the largest weight).

[0062] All of the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here.

[0063] The following are embodiments of the apparatus disclosed herein, which can be used to execute embodiments of the method disclosed herein. For details not disclosed in the apparatus embodiments of this disclosure, please refer to the embodiments of the method disclosed herein.

[0064] Figure 3 This is a schematic diagram of a face recognition model performance evaluation device provided in an embodiment of this disclosure.

[0065] like Figure 3 As shown, the performance evaluation device for this face recognition model includes:

[0066] The acquisition module 301 is configured to acquire a first image test set, a second image test set, and a third image test set.

[0067] The first calculation module 302 is configured to calculate the face comparison index of the face recognition model based on the first image test set;

[0068] The second calculation module 303 is configured to calculate the closed-set face retrieval index of the face recognition model based on the second image test set.

[0069] The third calculation module 304 is configured to calculate the open set face retrieval index of the face recognition model based on the third image test set.

[0070] Evaluation module 305 is configured to evaluate the performance of the face recognition model based on face comparison metrics, closed-set face retrieval metrics, and open-set face retrieval metrics.

[0071] The first, second, and third image test sets each contain multiple identifiers and multiple images under each identifier. Each identifier in the first image test set has no background image, each identifier in the second image test set has one background image, and each identifier in the third image test set with a preset ratio has one background image, while other identifiers have no background images (identifiers with a ratio one minus the preset ratio in the third image test set have no background images).

[0072] An identifier represents a person, and the multiple images associated with that identifier are multiple images of that person. The base image is a special image; for example, in a work attendance system, when each employee registers at the attendance machine, the image collected by the machine is the employee's base image.

[0073] Each identifier in the second image test set has a background image, so the second image test set can also be called the closed set test set. The index calculated based on the second image test set is called the closed set face retrieval index. In the third image test set, some identifiers have a background image, while others do not. Therefore, the third image test set can also be called the open set test set. The index calculated based on the third image test set is called the open set face retrieval index.

[0074] Optionally, the first calculation module 302 is also configured to calculate the corresponding indicators based on the first image test set, the second image test set, and the third image test set, respectively. This process can also be understood as generating three tasks: a task to calculate the face comparison indicator, a task to calculate the closed-set face retrieval indicator, and a task to calculate the open-set face retrieval indicator; and then executing the three tasks to obtain the face comparison indicator, the closed-set face retrieval indicator, and the open-set face retrieval indicator.

[0075] Obviously, the technical solution provided in this disclosure can be applied to any image processing technology field. In other words, the technical solution provided in this disclosure can evaluate the performance of any image recognition model. It only requires replacing the face comparison metrics, closed-set face retrieval metrics, and open-set face retrieval metrics with metrics for the corresponding image types.

[0076] According to the technical solution provided in this disclosure, a first image test set, a second image test set, and a third image test set are obtained; based on the first image test set, a face comparison index of the face recognition model is calculated; based on the second image test set, a closed-set face retrieval index of the face recognition model is calculated; based on the third image test set, an open-set face retrieval index of the face recognition model is calculated; and based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index, the performance of the face recognition model is evaluated. Therefore, by adopting the above technical means, the problem of not being able to accurately evaluate the performance of the face recognition model in the prior art can be solved, thereby improving the accuracy of evaluating the performance of the face recognition model.

[0077] Optionally, the first calculation module 302 is further configured to, during the process of using a face recognition model to identify whether any two images in the first image test set belong to the same identifier, count the number of false passes, the number of correct passes, the number of false rejections, the total number of any two images not belonging to the same identifier, and the total number of any two images belonging to the same identifier; calculate the false pass rate based on the number of false passes and the total number of any two images not belonging to the same identifier; calculate the correct pass rate based on the number of correct passes and the total number of any two images belonging to the same identifier; calculate the false rejection rate based on the number of false rejections and the total number of any two images belonging to the same identifier; and calculate the face comparison index based on the false pass rate, the correct pass rate, and the false rejection rate.

[0078] In this embodiment of the disclosure, given any two images, the system identifies whether the two images belong to the same identifier, that is, to determine whether the two images depict the same person. The determination result includes the following four cases:

[0079] It's actually the same person. They are not actually the same person. Judged to be the same person Passed correctly Error passed It was determined that they were not the same person. Error rejection Correct Rejection

[0080] Throughout the entire discrimination process, the following statistics are recorded: the number of incorrectly passed images, the number of correctly passed images, the number of incorrectly rejected images, the total number of any two images that do not belong to the same identifier, and the total number of any two images that belong to the same identifier.

[0081] The false pass rate can be obtained by dividing the number of false passes by the total number of times any two images do not belong to the same identifier; the correct pass rate can be obtained by dividing the number of correct passes by the total number of times any two images belong to the same identifier; and the false rejection rate can be obtained by dividing the number of false rejections by the total number of times any two images belong to the same identifier.

[0082] Optionally, the first calculation module 302 is further configured to determine a first threshold corresponding to a false pass rate of 1Ek, calculate a security index based on the first threshold and the correct pass rate, where 1Ek is scientific notation, representing moving the decimal point after 1 to the right by k places, and k is an integer greater than or equal to 2; determine a second threshold corresponding to a false rejection rate of 1Ek, and calculate a pass rate index based on the second threshold and the false pass rate; wherein, the face comparison index includes: a security index and a pass rate index.

[0083] A first threshold is determined when the false pass rate is 1Ek. A security index is calculated based on the first threshold and the correct pass rate. This can be understood as evaluating the correct pass rate when the false pass rate is low, and the evaluation result is the security index. A second threshold is determined when the false rejection rate is 1Ek. A passability index is calculated based on the second threshold and the false pass rate. This can be understood as evaluating the false pass rate when the false rejection rate is low, and the evaluation result is the passability index.

[0084] The correct pass rate is denoted as TPR, the false pass rate as FPR, and the false rejection rate as FRR. The calculation process described above can be expressed as follows:

[0085] TPR@FPR=1Ek(k=2, 3, 4, 5, 6...)

[0086] FPR@FRR=1Ek(k=2, 3, 4, 5, 6...)

[0087] k can be set by yourself.

[0088] Optionally, the second calculation module 303 is further configured to, during the process of retrieving the background image corresponding to any image in the second image test set using the face recognition model, count the number of images whose similarity to their corresponding background image ranks within the top N of all similarities, the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and whose similarity to their corresponding background image is greater than a third threshold, and the total number of pairs of images and background images; calculate a first pass index based on the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and the total number of pairs of images and background images; calculate a second pass index based on the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and whose similarity to their corresponding background image is greater than a third threshold, and the total number of pairs of images and background images; wherein, images and their background images under the same identifier have a corresponding relationship, and the closed-set face retrieval index includes: the first pass index and the second pass index.

[0089] During the process of retrieving the background image corresponding to any image in the second image test set using a face recognition model, the face recognition model calculates a similarity score between the image and the background image each time it performs a retrieval (i.e., compares any image with any background image). N is a positive integer and can be set by the user. The total number of pairs of images and background images can be understood as the number of pairs that can be formed between any image and any background image.

[0090] The first pass criterion can be obtained by dividing the number of images whose similarity to their corresponding base image ranks within the top N of all similarity scores by the total number of pairs of images and base images. The second pass criterion can be obtained by dividing the number of images whose similarity to their corresponding base image ranks within the top N of all similarity scores and whose similarity to their corresponding base image is greater than the third threshold by the total number of pairs of images and base images.

[0091] Optionally, the third calculation module 304 is further configured to, during the process of retrieving the background image corresponding to any image in the third image test set using the face recognition model, count the number of false passes, the number of correct passes, the number of correct rejections, the number of identifiers with background images, and the number of identifiers without background images; calculate the false pass rate based on the number of false passes and the number of identifiers without background images; calculate the correct pass rate based on the number of correct passes and the number of identifiers with background images; calculate the recognition success rate based on the number of correct passes, the number of correct rejections, the number of identifiers with background images, and the number of identifiers without background images; and calculate the open-set face retrieval index based on the false pass rate, the correct pass rate, and the recognition success rate.

[0092] When using a face recognition model to retrieve the background image corresponding to any image in the third image test set, the following four situations exist:

[0093] This person exists in the base map This person does not have a base map. This person was found Successful retrieval (passed correctly) Error retrieval (error passed) This person was not found. Error rejection Correct Rejection

[0094] The numbers of false passes, correct passes, and correct rejections are similar to the "false pass rate, correct pass rate, and false rejection rate" mentioned above. The concepts with the same names are similar, but the scenarios are different, so they will not be elaborated on here.

[0095] The false pass rate can be obtained by dividing the number of false passes by the number of identifiers without a base map; the correct pass rate can be obtained by dividing the number of correct passes by the number of identifiers with a base map; and the recognition success rate can be obtained by dividing the sum of the number of correct passes and the number of correct rejections by the sum of the number of identifiers with a base map and the number of identifiers without a base map.

[0096] Optionally, the third calculation module 304 is further configured to determine the first threshold corresponding to the value of the false pass rate as 1Ek, calculate the security index based on the first threshold and the correct pass rate, where 1Ek is scientific notation, representing moving the decimal point after 1 to the right by k places, and k is an integer greater than or equal to 2; calculate the success index based on the first threshold and the recognition success rate; wherein, the open-set face retrieval index includes: the security index and the success index.

[0097] The success index is calculated based on the first threshold and the recognition success rate. This can be understood as evaluating the recognition success rate when the false pass rate is low, and the evaluation result is the success index.

[0098] The correct pass rate is denoted as TPR, the incorrect pass rate as FPR, and the success rate indicator as ACC. The above calculation process can be expressed as follows:

[0099] TPR@FPR=1Ek(k=2, 3, 4, 5, 6...)

[0100] ACC@FPR=1Ek(k=2, 3, 4, 5, 6...)

[0101] Optionally, the first calculation module 302 is further configured to process the first image test set, the second image test set, and the third image test set as follows: estimate the image quality using a common face quality assessment algorithm, and directly remove low-quality images to improve the accuracy of comparison or retrieval. During this process, the effectiveness of image filtering is evaluated: all images in each test set are ranked by quality, a certain proportion of low-quality images are removed, and the remaining images are used to calculate metrics; the greater the improvement in metrics, the better the model performance.

[0102] Optionally, the evaluation module 305 is also configured to perform a weighted summation of the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index according to the target weight, and use the summation result as the performance index of the face recognition model.

[0103] It should be noted that the target weights are different in different scenarios. For example, in the face comparison scenario, the face comparison index has the largest weight; in the attendance tracking scenario where security requirements are not high, the closed-set face retrieval index has the largest weight; and in the attendance tracking scenario where security requirements are high, the open-set face retrieval index has the largest weight (because in the attendance tracking scenario where security requirements are not high, almost everyone who is clocking in has already registered on the clock-in machine, and there are few people who are faking attendance, so the closed-set face retrieval index has the largest weight).

[0104] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this disclosure.

[0105] Figure 4 This is a schematic diagram of the electronic device 4 provided in an embodiment of this disclosure. Figure 4 As shown, the electronic device 4 of this embodiment includes: a processor 401, a memory 402, and a computer program 403 stored in the memory 402 and executable on the processor 401. When the processor 401 executes the computer program 403, it implements the steps in the various method embodiments described above. Alternatively, when the processor 401 executes the computer program 403, it implements the functions of each module / unit in the various device embodiments described above.

[0106] Electronic device 4 can be a desktop computer, laptop, handheld computer, cloud server, or other electronic device. Electronic device 4 may include, but is not limited to, processor 401 and memory 402. Those skilled in the art will understand that... Figure 4 This is merely an example of electronic device 4 and does not constitute a limitation on electronic device 4. It may include more or fewer components than shown, or different components.

[0107] The processor 401 may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0108] The memory 402 can be an internal storage unit of the electronic device 4, such as a hard disk or RAM of the electronic device 4. The memory 402 can also be an external storage device of the electronic device 4, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, Flash Card, etc., equipped on the electronic device 4. The memory 402 can also include both internal and external storage units of the electronic device 4. The memory 402 is used to store computer programs and other programs and data required by the electronic device.

[0109] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0110] If an integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program may include computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. A computer-readable medium may include: any entity or device capable of carrying computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in a computer-readable medium may be appropriately added to or subtracted according to the requirements of legislation and patent practice in a jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media may not include electrical carrier signals and telecommunication signals.

[0111] The above embodiments are only used to illustrate the technical solutions of this disclosure, and are not intended to limit it. Although this disclosure has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this disclosure, and should all be included within the protection scope of this disclosure.

Claims

1. A method for evaluating the performance of a face recognition model, characterized in that, include: Obtain the first image test set, the second image test set, and the third image test set; Based on the first image test set, calculate the face comparison index of the face recognition model; Based on the second image test set, calculate the closed-set face retrieval index of the face recognition model; Based on the third image test set, calculate the open set face retrieval index of the face recognition model; The performance of the face recognition model is evaluated based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index. Based on the second image test set, the closed-set face retrieval index of the face recognition model is calculated, including: In the process of retrieving the background image corresponding to any image in the second image test set using the face recognition model, the following statistics are counted: the number of images whose similarity to their corresponding background image ranks within the top N of all similarity scores; the number of images whose similarity to their corresponding background image ranks within the top N of all similarity scores and whose similarity to their corresponding background image is greater than the third threshold; and the total number of pairs of images and any background image. The first pass metric is calculated based on the number of images and their corresponding base images that rank within the top N of all similarity scores, and the total number of pairs of images and base images. The second pass index is calculated by taking the number of images whose similarity to their corresponding base image ranks within the top N of all similarity scores and whose similarity to their corresponding base image is greater than the third threshold, as well as the total number of pairs of any image with any base image. Among them, there is a corresponding relationship between the images under the same identifier and their base images. The closed-set face retrieval index includes: the first pass index and the second pass index.

2. The method according to claim 1, characterized in that, The first image test set, the second image test set, and the third image test set each contain multiple identifiers and multiple images under each identifier. Each identifier in the first image test set has no background image, each identifier in the second image test set has one background image, and identifiers in the third image test set with a preset ratio have one background image, while other identifiers have no background image.

3. The method according to claim 1, characterized in that, Based on the first image test set, the face comparison index of the face recognition model is calculated, including: In the process of using the face recognition model to identify whether any two images in the first image test set belong to the same identifier, the number of incorrect passes, the number of correct passes, the number of incorrect rejections, the total number of any two images not belonging to the same identifier, and the total number of any two images belonging to the same identifier are counted. Calculate the error pass rate based on the number of failed passes and the total number of any two images that do not belong to the same identifier; The pass rate is calculated based on the number of correctly passed images and the total number of images that belong to the same identifier. The false rejection rate is calculated based on the number of false rejections and the total number of any two images belonging to the same identifier. The face comparison index is calculated based on the false pass rate, the correct pass rate, and the false rejection rate.

4. The method according to claim 3, characterized in that, The face comparison index is calculated based on the false pass rate, the correct pass rate, and the false rejection rate, including: A first threshold is determined when the error pass rate is 1E-k, and a security index is calculated based on the first threshold and the correct pass rate, where k is an integer greater than or equal to 2; A second threshold is determined when the false rejection rate is 1E-k, and a passability index is calculated based on the second threshold and the false pass rate; The face comparison indicators include: the security indicators and the passability indicators.

5. The method according to claim 1, characterized in that, Based on the third image test set, the open-set face retrieval index of the face recognition model is calculated, including: During the process of using the face recognition model to retrieve the background image corresponding to any image in the third image test set, the number of incorrect passes, the number of correct passes, the number of correct rejections, the number of identifiers with background images, and the number of identifiers without background images are counted. Calculate the error pass rate based on the number of error passes and the number of identifiers that do not have a base map; Calculate the pass rate based on the number of correct passes and the number of identifiers with base maps; The recognition success rate is calculated based on the number of correct passes, the number of correct rejections, the number of identifiers with a base map, and the number of identifiers without a base map. The open-set face retrieval index is calculated based on the error pass rate, the correct pass rate, and the recognition success rate.

6. The method according to claim 5, characterized in that, The open-set face retrieval index is calculated based on the false pass rate, the correct pass rate, and the recognition success rate, including: A first threshold is determined when the error pass rate is 1E-k, and a security index is calculated based on the first threshold and the correct pass rate, where k is an integer greater than or equal to 2; Calculate the success index based on the first threshold and the recognition success rate; The open-set face retrieval metrics include: the security metrics and the success metrics.

7. A device for evaluating the performance of a face recognition model, characterized in that, include: The acquisition module is configured to acquire the first image test set, the second image test set, and the third image test set. The first calculation module is configured to calculate the face comparison index of the face recognition model based on the first image test set; The second calculation module is configured to calculate the closed-set face retrieval index of the face recognition model based on the second image test set. The third calculation module is configured to calculate the open-set face retrieval index of the face recognition model based on the third image test set. The evaluation module is configured to evaluate the performance of the face recognition model based on the face comparison index, the closed-set face retrieval index, and the open-set face retrieval index. The second calculation module is specifically configured to: during the process of retrieving the background image corresponding to any image in the second image test set using the face recognition model, count the number of images whose similarity to their corresponding background image ranks within the top N of all similarities, the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and whose similarity to their corresponding background image is greater than a third threshold, and the total number of pairs of images and background images; calculate a first pass index based on the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and the total number of pairs of images and background images; calculate a second pass index based on the number of images whose similarity to their corresponding background image ranks within the top N of all similarities and whose similarity to their corresponding background image is greater than a third threshold, and the total number of pairs of images and background images; wherein, images and their background images under the same identifier have a corresponding relationship, and the closed-set face retrieval index includes: the first pass index and the second pass index.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 6.

9. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 6.

Images