An algorithm evaluation system and detection method for performance detection of human identity verification devices

Abstract

The present invention discloses an algorithm evaluation system and a detection method for the performance detection of human identity verification devices. The overall algorithm evaluation system is jointly composed of a database call module, a data preprocessing module, a device interface debugging module, an automated test module, a quality evaluation module, a dataset management module, an image algorithm module, and an information statistics module. Based on this algorithm evaluation system and detection method, a large-scale face test database and a preprocessed face database collected on-site by the device can be called according to the configuration rules, and face data is sequentially pushed to the device under test for face recognition. The test results such as feature extraction and feature comparison of the device under test are controlled and obtained through the test function interface, realizing the detection of face recognition performance indicators that can take into account both software and hardware factors of human identity verification devices.

Description

Technical Field

[0001] The present invention relates to a human identity verification device or a performance index detection technology of a human identity verification device, and specifically relates to a detection technology for the "false acceptance rate (FAR) and false rejection rate (FRR)" indicators of the face recognition performance of a human identity verification device or a human identity verification device. Background Art

[0002] As one of the most common biometric recognition modes in the field of biometric recognition, face recognition technology has been widely applied in multiple industrial fields in recent years. A human identity verification device that compares an identified object with its ID photo or / and the electronic photo in the chip of the ID card held by it based on face recognition technology to verify whether the person to be verified is consistent with the information on the ID card held by him or her or the identity information he or she claims is a typical device type in many face recognition applications. It is widely used in occasions such as the identity verification of inbound and outbound passengers, the identity verification of visitors to important places, the identity verification of hotel check-in guests, the identity verification of financial business in the financial industry, the identity verification of social security personnel, and the identity verification of examination candidates, etc., which require personnel identity verification and management control, so as to achieve the identity verification of relevant personnel and achieve the purpose of security prevention.

[0003] Due to the particularity of the actual use environment, users have relatively high expectations for the face recognition performance of such devices. Therefore, relatively high requirements are also put forward for the face recognition technology used in such devices. Driven by demand, the scale of the biometric recognition market in China has grown rapidly, and research and development enterprises and institutions related to face recognition products have emerged continuously. The quality of the produced human identity verification device products is also uneven. Therefore, effective means are urgently needed to reasonably evaluate the face recognition performance of such devices. Summary of the Invention

[0004] The present invention designs an algorithm evaluation system for detecting the face recognition performance index of a human identity verification device, and gives a detection method for detecting the face recognition performance index of a human identity verification device using this algorithm evaluation system. Based on the algorithm evaluation system and detection method given by the present invention, the face recognition performance of a human identity verification device can be effectively and accurately tested and evaluated.

[0005] The algorithm evaluation system and detection method designed by the present invention are applicable to various devices for human identity verification through face recognition technology, and are applicable to both the device types for comparing the information of the verified person with the information on the certificate he / she holds and the device types for comparing the information of the verified person with the claimed identity information. Among them, the information on the certificate held includes all certificates associated with the face photos of relevant personnel, such as the second-generation resident identity card, passport, social security card, people's police officer's certificate, student ID card, work permit, visitor's permit, etc.; the claimed identity information includes all information that can trace the face photos of relevant personnel, such as the name of the person, ID number, social security card number, people's police officer number, student ID number, work permit number, visitor's permit number, etc.

[0006] The algorithm evaluation system for detecting the performance of human identity verification devices designed by the present invention includes a database call module, a data preprocessing module, a device interface debugging module, an automated test module, a quality evaluation module, a dataset management module, and an image algorithm module;

[0007] The device interface debugging module can debug the interface functions of the device under test through the API dynamic link library file and the algorithm configuration file provided by the device manufacturer under test, and is used to push the face images in the target set and the probe set to the device under test to run the face recognition algorithm during the automated testing process of the automated detection tool module, and obtain the test results;

[0008] The test database call module, the data preprocessing module, and the dataset management module cooperate to form the data source of the system, and can be used to download a large-scale face test database and / or collect and preprocess the on-site face test database;

[0009] The automated test module is used to detect the performance indicators of "false acceptance rate (FAR) and false rejection rate (FRR)" by calling the face recognition algorithm running on the device under test in a computer or server environment;

[0010] The quality evaluation module can detect the face image quality evaluation performance indicators by calling the image algorithm module for the on-site live face images collected by the device under test, and can also simultaneously perform the same for the visualized face images of the certificates collected by the device under test.

[0011] In addition, the evaluation system also includes a function module for managing the user's document-level permissions, and the function module includes one or more of a user login module, a project registration module, and a user management module.

[0012] The evaluation system includes an information statistics module for data analysis and processing of the test results, and the information statistics module includes one or more of a test result module, a project management module, a project statistics module, and an algorithm statistics module.

[0013] The automated test module in the system cooperates with the device interface call module, data preprocessing module, database call module, and dataset management module to perform performance detection of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)" on the device under test.

[0014] The device interface call module obtains the face recognition algorithm test interface of the device under test; the data preprocessing module uses the face images collected on-site by the device under test as the on-site collection database in this test database after preprocessing; the data call module calls and downloads the database required for this test; the dataset management module sets up a data security mechanism to manage the total test database for this test, and the automated test module calls the above modules to achieve performance detection of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)".

[0015] The quality evaluation module cooperates with the device interface call module, data preprocessing module, and image algorithm module to evaluate the image quality of the face images collected on-site by the device under test.

[0016] The device interface call module obtains and uploads the live face images, ID photos, or visible face images of ID cards collected on-site by the device under test; the data preprocessing module performs tests such as face detection on the images obtained by the device interface call module; the image quality evaluation module evaluates the face image quality of the uploaded images by calling the image algorithm module, and the image algorithm module performs automated tests on the face images collected on-site exported by the device under test according to the set technical requirements. The quality evaluation module analyzes and processes the data generated by the image algorithm module to give a compliance evaluation.

[0017] The detection method for the performance detection of the human identity verification device provided by the present invention includes:

[0018] Using the downloaded large-scale face test database and the preprocessed on-site collection database of the device, they are sequentially pushed to the device under test according to the configuration rules for face algorithm operation;

[0019] Through the test function interface call, control and obtain the test results such as feature extraction and feature comparison of the device under test to achieve performance index detection;

[0020] The detection method includes the face image quality performance detection of the device under test, including:

[0021] Automatically test the live face images collected on-site by the image algorithm according to the set technical requirements, and then analyze and process through the quality evaluation module to obtain the image quality evaluation result;

[0022] Automatically test the visible face image of the certificate collected according to the set technical requirements through an image algorithm, and then analyze and process it through a quality evaluation module to obtain an image quality evaluation result.

[0023] In the detection method, download the test database according to the data security mechanism, and push it to the device under test in sequence according to the configuration rules for face algorithm operation, and realize the encryption and desensitization of face image data with reference to the mapping relationship. Ensure that the face data in the called face test database cannot be exported and reused in the device under test, thereby ensuring the safe use of the face data in the face test database.

[0024] Furthermore, the detection method includes specific detection steps for testing the performance indicators of the false acceptance rate (FAR) and false rejection rate (FRR) of various human identity verification devices.

[0025] The algorithm evaluation system and detection method designed by the present invention can realize the intelligent and automatic detection of the face image quality and the performance indicators of "false acceptance rate (FAR) and false rejection rate (FRR)" of human identity verification devices, reduce the technical ability requirements and labor cost investment of such detection work, and greatly improve the detection efficiency.

[0026] Moreover, while evaluating the performance of the face recognition algorithm of the human identity verification device, the detection method given by the present invention comprehensively considers the influence of the hardware facilities of the human identity verification device on the recognition performance. In specific inspection and detection applications, it can support test databases of tens of millions of levels and above, support multi-platform and multi-thread synchronous operation, and realize the intelligence and automation of the test process, the standardization and transparency of the test process, and the visualization and traceability of the test results on the premise of ensuring the safe use of the face information data in the test database. It not only greatly improves the detection efficiency, but also improves the fairness and reasonableness of the performance test results of the human identity verification device, which is of great significance for promoting the technological progress of the human identity verification device and promoting the further wide application of the human identity verification device.

[0027] In addition, the present algorithm evaluation system and detection method are applicable to various devices for human identity verification through face recognition technology, and are applicable to both the device types for comparing the consistency between the verified person and the information on the certificate he holds, and the device types for comparing the consistency between the verified person and the claimed identity information. Among them, the information on the certificate held includes all certificates associated with the face photos of the person, such as the second-generation resident ID card, passport, social security card, people's police ID card, student ID card, work ID card, visitor ID card, etc., and the claimed identity information includes all information that can trace the face photos of the associated person, such as the person's name, ID number, social security card number, people's police police number, student ID number, work ID number, visitor ID number, etc.

[0028] In addition, the algorithm evaluation system and detection method of the present invention are compatible with multiple types of operating systems, support multi-threaded operation and offline automated testing, and can be applied to the detection of various products that use face biometrics for identity authentication, such as human identity verification devices in hotels, face recognition devices at entrances and exits, face recognition access control integrated machines, face recognition turnstiles, face recognition bayonet cameras, face recognition attendance machines, and face recognition payment systems. The detection method based on the algorithm evaluation system of the present invention can greatly simplify the test operation process and significantly improve the detection efficiency of human identity verification devices while ensuring the scientific and fair test results of the performance of human identity verification devices to the greatest extent. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The present invention will be further described below in conjunction with the accompanying drawings and specific embodiments.

[0030] Figure 1 Schematic structural diagram of the algorithm evaluation system of the present invention;

[0031] Figure 2 Logic relationship diagram of the algorithm evaluation system of the present invention;

[0032] Figure 3 Flowchart of the corresponding detection method of the algorithm evaluation system of the present invention;

[0033] Figure 4 Flowchart of the test for face image quality performance indicators in the detection method of the present invention;

[0034] Figure 5 Schematic diagram of the face data security mechanism corresponding to the algorithm test system of the present invention;

[0035] Figure 6 Flowchart of the automated test for the performance indicators of false acceptance rate (FAR) and false rejection rate (FRR) in the detection method of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0036] In order to make the technical means, creative features, achieved purposes and effects realized by the algorithm evaluation system and detection method designed by the present invention easy to understand, the present invention will be further described below in conjunction with specific illustrations.

[0037] As the scale of the open test library of face recognition algorithms is getting larger and larger, combined with the actual use of the device, the scale of the test library generally reaches the level of ten thousand and above. Therefore, it is difficult to implement a large-scale test method with real people for device performance testing, and the existing test scheme of 20 real-person tests cannot adapt to the efficient and effective performance detection of human identity verification devices or 1:1 human identity verification devices.

[0038] Therefore, the present invention designs an algorithm evaluation system that takes into account both the algorithm and the hardware performance of the human and certificate verification device, and a detection method for evaluating the face recognition performance indicators based on this system.

[0039] Figure 1 It is a schematic structural diagram of the algorithm evaluation system for detecting the performance indicators of the human and certificate verification device of the present invention.

[0040] As Figure 1 shown, this system is based on the B / S architecture and realizes the evaluation function through web access operations. The system consists of a test operation terminal (PC terminal) 100 and a test server (SERVER terminal) 200, and is implemented in cooperation with a storage server (SERVER terminal) 300.

[0041] In some specific embodiments, the test operation terminal (PC terminal) 100 is composed of a corresponding PC, which accesses the test server (SERVER terminal) 200 in the form of a web and provides a test operation WEB interface for users. At the same time, on the test WEB operation interface (PC terminal) 100, there are functional modules such as a database call module 110, a data preprocessing module 120, a device interface debugging module 130, an automated test module 140, a quality evaluation module 150, a test result module 160, a user login module 170, a project registration module 180, etc. Through these functional modules, users can perform test operations on the device to be tested on the visual interface to achieve the detection of relevant performance indicators.

[0042] In cooperation with it, in the test server (SERVER terminal) 200, there are functional modules such as a user management module 210, a project management module 220, a dataset management module 230, an image algorithm module 240, a project statistics module 250, and an algorithm statistics module. This test server (SERVER terminal) 200 can be used to interact with the storage server (SERVER terminal) 300 storing the face test database to achieve the integrated management of the dataset and the detection process.

[0043] Figure 2 It is a logical relationship diagram of the algorithm evaluation system of the present invention.

[0044] As Figure 2 shown, the device interface debugging module 130 in this example solution is located on the test WEB interface (PC terminal) 100 and is used to debug the test interface with the device to be tested 400. This device interface debugging module 130 can call the test interface function through the API dynamic link library provided by the device manufacturer and the algorithm configuration file, and dock with the automated test module to ensure that the face images in the target set and the detection set are pushed to the device to be tested to run the face recognition algorithm and obtain the test results.

[0045] In this solution, the database call module 110 and the data preprocessing module 120 run on the test WEB interface (PC side) 100, while the dataset management module 230 runs on the test server (SERVER side) 200. At the same time, the three modules cooperate with each other to form the test database source of the algorithm evaluation system.

[0046] As an example, the test database source here can include the downloaded large-scale face test database and the on-site collected face database. Among them, the large-scale face test database is obtained by interacting and downloading through the test server 200 and the storage server (SERVER side) 300; the on-site collected face database is obtained by calling the test interface between the PC side 100 and the device under test 400.

[0047] Among them, the face data of the on-site collected face database is the face images of the test personnel collected on-site using the device under test and preprocessed by the data preprocessing module 120.

[0048] The database call module 110 prepares a dataset downloaded in batches from the storage server according to the configuration rules for this detection, which includes a target set and a probe set.

[0049] The dataset management module 230 can manage the dataset formed by the face images collected on-site prepared by the data preprocessing module. In a specific example, the dataset management module 230 downloads the corresponding dataset from the test database according to the data scale confirmed by the detection requirements of the current time, aggregates the downloaded dataset and the on-site collected dataset, and centrally processes them with the single-person dataset as the unit.

[0050] As an example, a data security mechanism is also adopted between these three functional modules to realize the access rights and supervision and control of the test database, and can feedback on the dataset anomalies in the test database and upload new datasets during the automated test process.

[0051] The automated test module 140 in this solution runs on the PC side and docks with the database call module 110 and the device interface debugging module 130 to detect the performance indicators of the false acceptance rate (FAR) and false rejection rate (FRR) of the device under test.

[0052] Among them, the device interface debugging module 130 is used to debug the interface communication between the algorithm evaluation system and the device under test, and sequentially call the corresponding interface functions of the face recognition algorithm of the device under test using the test program according to the test steps set by the automated test module 140 to realize data interaction. As an example, the interface functions here at least include functions such as initializing the algorithm, obtaining the feature length, feature extraction, feature comparison and returning the similarity, querying, releasing algorithm resources, and obtaining version information.

[0053] The automated test module 140, in cooperation with the dataset management module 230, respectively passes the face images in the single-person dataset corresponding to the target set and the detection set through the device interface debugging module 130 to sequentially call different interface functions according to the test program to obtain the algorithm operation data of the device under test, and processes the obtained results according to the performance test method of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)", so as to realize the detection of the performance indicators of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)".

[0054] The quality evaluation module 150 in this solution runs on the PC side. It docks and cooperates with the data preprocessing module 120 and the image algorithm module 240 running on the test server (SERVER side) 200, and the image algorithm module 240 performs quality evaluation on the on-site live face images and document visual face images collected by the device under test.

[0055] The image algorithm module 240 here is used to perform compliance tests on face images according to the relevant technical requirements in the corresponding standards, including indicators such as image format, face occlusion, interpupillary distance, pose, and face area.

[0056] As an example, the image algorithm module 240 is used to perform automated tests on the face images collected on-site exported by the device under test according to the set standard technical requirements, and the quality evaluation module 150 analyzes and processes the data generated by the image algorithm module 240 to give a compliance evaluation.

[0057] As an example, (1) the image algorithm module performs automated tests on the on-site live face images collected on-site and exported by the device under test, and gives parameters such as image format, occlusion, interpupillary distance, pose, and face area for the face recognition images in the detection set; the quality evaluation module obtains these parameters and makes a judgment according to the corresponding indicators specified in GB / T35678-2017;

[0058] (2) The image algorithm module performs automated tests on the on-site visual face images collected on-site and exported by the device under test, and gives parameters such as image format, occlusion, interpupillary distance, pose, and face area for the face recognition images in the target set; the quality evaluation module obtains these parameters and makes a judgment according to the corresponding indicators specified in GB / T35678-2017;

[0059] (3) The image algorithm module performs automated tests on the on-site ID photos collected on-site and exported by the device under test, and gives parameters such as image format, occlusion, interpupillary distance, pose, and face area for the face recognition images in the target set; the quality evaluation module obtains these parameters and makes a judgment according to the corresponding indicators specified in GB / T 35678-2017.

[0060] The user login module 180 and the project registration module 170 in this solution run on the PC side, and are docked and coordinated with the user management module 250 and the project management module 220 running on the test server (SERVER side) 200, and are used for the system to manage the hierarchical permissions of users according to conditions such as projects, algorithms, and test situations.

[0061] As an example, this user management module 250 can be used to manage the users of the test system and allocate usage permissions, including users such as super administrators, data administrators, and test engineers.

[0062] Users with different permissions use the user login module 180 to enter the test system, and different modules are used. For example, the super administrator has the highest authority and can access and use all modules; the data administrator manages the data set and can access management modules related to data such as the data preprocessing module, the database call module, the data set management module, and the project management module; the test engineer judges the image quality and conducts performance tests on "false acceptance rate (FAR) and false rejection rate (FRR)", and can access the data preprocessing module, the database call module, the image algorithm module, the quality judgment module, the device interface call module, and the automated test module.

[0063] The project management module 220 manages the testing of each device under test, which is managed by the super administrator, and the test engineer can only view it. Project login management is for the test engineer to create a project based on the device under test, start the test task, and conduct image quality judgment and performance tests on "false acceptance rate (FAR) and false rejection rate (FRR)".

[0064] The test result module 160 in this solution runs on the PC side, and is docked and coordinated with the project statistics module 250 and the algorithm statistics module 260 running on the test server (SERVER side) 200, and is used for the data analysis and processing of test results, and conducts classified hierarchical statistical management according to different projects and algorithms.

[0065] As an example, this test result module 160 summarizes and manages the test data obtained by each device under test through the automated test module and the image quality judgment module.

[0066] The project statistics module 250 manages by taking the device under test as a unit, and the number of times of image quality judgment and performance tests on "false acceptance rate (FAR) and false rejection rate (FRR)" with different requirements for each device under test is used as the test task. The algorithm statistics module 260 manages the test data of each image quality judgment and performance test on "false acceptance rate (FAR) and false rejection rate (FRR)" by taking the test task as a unit. In this solution, the data of project statistics management and algorithm statistics management come from the test result module and are distinguished according to the different units above.

[0067] According to the algorithm evaluation system and detection method for the performance detection of human identity verification devices given by the present invention, users can conveniently use the downloaded large-scale face test database and the preprocessed on-site acquisition database of the device to be pushed to the device to be tested in sequence according to the configuration rules for face algorithm operation, and then control and obtain the test results such as feature extraction and feature comparison of the device to be tested through the test function interface, so as to realize the detection of performance indicators such as face image quality, false acceptance rate (FAR), and false rejection rate (FRR). Thus, the face recognition performance of human identity verification devices can be effectively and efficiently evaluated.

[0068] When using the present algorithm evaluation system and detection method to detect the performance indicators of false acceptance rate (FAR) and false rejection rate (FRR) for human identity verification devices, the specific operation steps and implementation plans are as follows:

[0069] 1) Initialize the algorithm evaluation system, start the storage server, test server and connect them to the test computer;

[0070] 2) Check the dll file and algorithm configuration file of the device to be tested provided by the manufacturer and start the device to be tested;

[0071] 3) Query and record the preset parameters such as the face recognition comparison threshold of the device to be tested;

[0072] 4) According to the performance level requirements of "false acceptance rate (FAR) and false rejection rate (FRR)" of the device to be tested, select the scale of the test database, confirm the required number of test personnel, that is, the number of non-repeating test personnel in the target set and the number of test face images in the probe set;

[0073] 5) Based on the scale of the test database, determine the number of on-site acquisition data sets and the number of data sets in the downloaded face test database according to the ratio rule.

[0074] 6) Use the data preprocessing module to prepare the on-site acquired face data set;

[0075] 7) Use the database call module to download the data set from the face test database according to the configuration rule of the sample distribution requirements specified by the standard;

[0076] 8) The test system establishes a connection with the device to be tested, uses the device interface call module to debug the interface to ensure the successful call of the interface function and data interaction;

[0077] 9) Start the automated test module and cooperate with the device interface call module and the data set management module to complete the preparations before the test;

[0078] 10) Conduct the first stage of testing: Feature extraction. The automated testing module drives the initialization of the device interface call module, calls the feature extraction interface function of the device under test, and performs face image feature extraction on each face image in the target set and the probe set in the test database managed by the dataset management module. The implementation of feature extraction is carried out by the face recognition algorithm of the device under test. The automated testing module analyzes and processes information such as the face feature file library, the total number of samples, the number of successful extractions, and their corresponding identity identifiers for both the target machine and the probe set, verifies the corresponding relationship between the two datasets, and records and stores them;

[0079] 11) Conduct the second stage of testing: Feature comparison. The automated testing module drives the initialization of the device interface call module, calls the feature comparison interface function of the device under test, and performs feature comparison on the feature data of the probe set and the feature data of the target set through the face recognition algorithm of the device under test. The automated testing module obtains the feature comparison results and analyzes and processes the similarity value accurate to 0.0001 for each comparison and its corresponding data information;

[0080] 12) Conduct the third stage of testing: Calculate the performance test results of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)". The automated testing module comprehensively analyzes the obtained data, calculates "FAR" and "FRR" according to the formula, conducts compliance evaluation according to the standard level requirements, and obtains the performance test results and the ROC curve of the device under test.

[0081] Next, for the algorithm evaluation system for detecting the performance of the human identity verification device given in this example, the implementation process of detecting the performance indicators of the human identity verification device will be illustrated by an example.

[0082] Figure 3 The following shows the implementation process of detecting the performance indicators of the human identity verification device based on the algorithm evaluation system in this example. It mainly includes the following steps:

[0083] (1) Initialize the algorithm evaluation system according to user operations.

[0084] (2) Log in to the system using the corresponding user account according to user permission requirements.

[0085] (3) Create a new project, input the manufacturer information and device information of the device under test, and upload the algorithm configuration file and relevant technical materials of the device; at the same time, historical records can be retrieved according to the manufacturer and device name, and project information can be automatically filled in.

[0086] (4) Test interface debugging: select the device's dynamic link library and algorithm configuration file to verify whether the interface of the device under test complies with the test interface requirements specified in relevant industry standards and specifications such as the "General Technical Requirements for Identity Verification Equipment for Security Face Recognition Applications". For example, this includes at least the initialization algorithm, feature length acquisition, feature extraction, feature comparison and return of similarity, query, and algorithm resource and version information acquisition functions.

[0087] (5) According to the requirements of the performance indicators to be tested, if it is a performance test of face image quality, proceed to step (6); if it is a performance test using false acceptance rate (FAR) and false rejection rate (FRR), proceed to step (7).

[0088] (6) The algorithm evaluation system performs facial image quality detection based on the performance index requirements of the identity verification equipment by using the data preprocessing module and quality evaluation module located on the PC side and the image algorithm module located on the test server (SERVER side).

[0089] (7) The detection system performs false acceptance rate (FAR) and false rejection rate (FRR) detection by using an automated test module and a test result module located on the PC side according to the performance index requirements related to the human body security inspection equipment.

[0090] (8) After completing each test item according to the test requirements, generate the test results, calculate and compile statistics to form a report and upload it to the storage server.

[0091] (9) Review any data anomalies that may occur in the downloaded test database during the test process and report them to the storage server; review the on-site collection database and upload it to the storage server after confirming that it is correct.

[0092] (10) Access the storage server and query the project and algorithm statistical results according to the device application requirements.

[0093] In a more preferred implementation, this detection scheme can be coordinated through functional modules such as the device interface calling module, data preprocessing module, image quality judgment module, image algorithm module, etc. in the algorithm evaluation system to achieve facial image quality detection of the device under test based on performance indicator requirements related to the identity verification equipment.

[0094] For example, during implementation, the device interface call module acquires the live face images, ID photos, or visible ID face images collected on-site from the device under test, and uploads the acquired images to the test server (SERVER side). In coordination, the data preprocessing module conducts tests such as face detection on the images acquired by the device interface call module. The image quality evaluation module calls the image algorithm module to evaluate the quality of the uploaded images (here, the uploaded images are directly the face images exported from the device under test. Specifically, during implementation, they can be exported through the software of the device under test, or directly transferred to the storage path of the test computer using a USB flash drive or network interface. The image algorithm module can directly access the face images in the storage folder), and outputs the results to the PC side.

[0095] Accordingly, in this example, an algorithm evaluation system is used to conduct test operations on the face image quality performance indicators of the person-certificate verification device, which is mainly implemented through the following steps (see Figure 4 ):

[0096] (6.1) Initialize the system. The PC (computer) operates the WEB interface to enter the data preprocessing module interface.

[0097] (6.2) Through the device interface debugging module, call the live face images collected on-site from the device under test, and upload them to the PC side in FTP mode to obtain the on-site live face images.

[0098] (6.2) Interface and run with the image algorithm module in the test server (SERVER side), start the image quality evaluation module, and evaluate the live face images collected on-site.

[0099] (6.3) Regarding one of the face image quality performance indicator requirements: the on-site live face images collected meet the relevant technical requirements of 4.2 in GB / T 35678-2017. The image quality evaluation module compares the acquired face images and determines whether the device meets this performance indicator requirement; and automatically generates and stores the test results of a single project and uploads them.

[0100] (6.4) Regarding the second face image quality performance indicator requirement: the visible ID face images collected meet the relevant technical requirements of 4.2 in GB / T 35678-2017. The image quality evaluation module compares the acquired face images and determines whether the device meets this performance indicator requirement. And automatically generates and stores the test results of a single project and uploads them.

[0101] In a more preferred embodiment, this detection solution can perform the performance detection of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)" on the device under test through the coordinated docking of functional modules such as the device interface call module, data preprocessing module, database call module, automated test module, and dataset management in the algorithm evaluation system.

[0102] Among them, the device interface call module is used to obtain the interfaces required for the face algorithm test of the device under test, including at least initializing the algorithm, obtaining the feature length, feature extraction, feature comparison and returning the similarity, querying, releasing algorithm resources, and obtaining version information, etc.

[0103] The data preprocessing module uses the face images collected on-site by the device under test as the on-site collection database in this test database. Among them, the test database includes the on-site collected dataset and the downloaded dataset; the on-site collected dataset is directly obtained from the device under test by the data preprocessing module; the downloaded dataset is obtained from the storage server by the database call module.

[0104] The data call module realizes the download of the test database in this test, and the scale can reach ten thousand levels and above. As an example, the face test library stored in the storage server can be directly accessed and downloaded by the database call module.

[0105] The dataset management module sets up a data security mechanism to manage the total test database of this test, such as Figure 5 shown.

[0106] The automated test module calls the above-mentioned device interface call module, data preprocessing module, database call module, and dataset management to dock and cooperate to achieve the "FAR and FRR" test.

[0107] Accordingly, the specific steps for this example to perform the performance index test of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)" on the human identity verification device through the algorithm evaluation system are as follows (see Figure 6 ):

[0108] (7.1) Query and record the preset parameters such as the face recognition comparison threshold of the device.

[0109] (7.2) Initialize the system and enter the database call module; select the test library with the corresponding data ratio and scale according to the grade requirements of the false acceptance rate and false rejection rate of the device.

[0110] (7.3) Use the dynamic link library (.dll file) provided by the manufacturer to debug the interface of the device under test through the device interface call module, and realize at least the interface calls such as initializing the algorithm, obtaining the feature length, feature extraction, feature comparison and returning the similarity, querying, releasing algorithm resources, and obtaining version information.

[0111] (7.3) Enter the data preprocessing module. For the face images collected on-site, perform data matching according to the requirements of the test library, with the proportion of on-site live face images in the target set being no less than 2%.

[0112] (7.4) Through the dataset management module, comprehensively obtain the test database for this project's test from the two-source databases.

[0113] (7.5) Through the device function interface call module, call the test interface function. For each sample image in the target set of the current test database stored on the PC side, push it to the device under test for feature value extraction. Sequentially call the feature extraction interface to obtain and save the face feature file library 1, record the total number of samples N in the target set, the total number of samples C1 with successful feature extraction, and the identity identification information of each sample image in the target set.

[0114] (7.6) Through the device function interface call module, call the test interface function. For each face image in the probe set of the current test database stored on the PC side, push it to the device under test for feature value extraction. Sequentially call the feature extraction interface to obtain and save the face feature file library 2, record the total number of samples M in the probe set, the total number of samples C2 with successful feature extraction, and the identity identification information of each face image in the probe set.

[0115] (7.7) Check the one-to-one correspondence management of the results in steps (7.5) and (7.6) to confirm that the quantity and identity identification are consistent.

[0116] (7.8) Through the device function interface call module, call the test interface function. Read the face feature file library 1 of the target set and the face feature file library 2 of the probe set into the memory. Compare the feature data of each face image in the probe set with the feature data of each sample image in the target set one by one by calling the feature comparison function. The face recognition algorithm runs on the device under test; record N×M comparison similarity degrees, and the similarity degree value ranges from 0.0 to 1.0 (the similarity degree value is accurate to 4 decimal places). The larger the similarity degree value, the higher the similarity level.

[0117] (7.9) Among the N×M recorded comparison similarity degrees, all the comparison similarity degrees of the same person are put into the similarity set S1, and all the comparisons of different people are put into the similarity set S2. The number of elements with similarity less than the threshold T in the similarity set S1 is the false rejection number, denoted as N R ; The number of elements with similarity greater than the threshold T in the similarity set S2 is the false acceptance number, denoted as N A ;

[0118] (7.10) The performance index requirements of "false acceptance rate FAR and false rejection rate FRR" are divided into two levels: basic level and enhanced level. Devices of different levels should meet the following requirements respectively:

[0119] a) Basic-level device: When the False Acceptance Rate (FAR) is less than or equal to 0.1% under the same set threshold condition, the False Rejection Rate (FRR) is less than or equal to 5%.

[0120] b) Enhanced-level device: When the False Acceptance Rate (FAR) is less than or equal to 0.01% under the same set threshold condition, the False Rejection Rate (FRR) is less than or equal to 5%.

[0121] Accordingly, count all the feature comparison results, and calculate and output the False Acceptance Rate (FAR) and False Rejection Rate (FRR) under the preset face comparison similarity threshold of the device according to formula (1) and formula (2) respectively. And automatically generate the test results of a single project and store and upload them, and judge whether the device meets the corresponding performance index requirements according to the calculation results.

[0122]

[0123]

[0124] Among them, when the feature extraction failure rates of both the target set and the detection set are 0%, C1 of the feature file library 1 = N, and C2 of the feature file library 2 = M; when the feature extraction failure rate is not 0%, then (N - C1) and (M - C2) are counted as the number of "feature extraction failures" and both are included in N A and N R .

[0125] As can be seen from the above example, the performance detection scheme of the human identity verification device given by the present invention can support a test database with a scale of tens of millions or more in specific inspection and detection applications, support multi-platform and multi-thread synchronous operation, and on the premise of ensuring the safe use of the face information data in the test database, it realizes the intelligence and automation of the test process, the standardization and transparency of the test process, and the visualization and traceability of the test results. It not only greatly improves the detection efficiency, but also improves the fairness and rationality of the performance test results of the human identity verification device, which is of great significance for promoting the technological progress of the human identity verification device and further promoting the wide application of the human identity verification device.

[0126] The method of the present invention, or a specific system unit, or a part of the unit, can be deployed on a hard disk, an optical disc, or an electronic device such as a smart phone or a computer through program code. When the electronic device loads the program code and executes the program code, the corresponding electronic device becomes a device for implementing the present invention. The method of the present invention and the test database information can also be transmitted through a network in the form of program code. When the program code is received, loaded and executed by an electronic device (such as a smart phone), the corresponding electronic device becomes a remote device for implementing the present invention.

[0127] The foregoing has shown and described the basic principles, main features and advantages of the present invention. Those skilled in the art should understand that the present invention is not limited by the above embodiments, and what is described in the above embodiments and the specification only illustrates the principles of the present invention. Without departing from the spirit and scope of the present invention, the present invention will have various changes and improvements, and these changes and improvements fall within the scope of the present invention claimed. The scope of the present invention claimed is defined by the appended claims and their equivalents.

Claims

1. An algorithm evaluation system for the performance detection of human identity verification devices, characterized in that, It includes a database call module, a data preprocessing module, a device interface debugging module, an automated testing module, a quality evaluation module, a dataset management module, and an image algorithm module; The device interface debugging module can debug the interface functions of the device under test through the API dynamic link library file and the algorithm configuration file provided by the device under test manufacturer, and is used to push the face images in the target set and the probe set to the device under test to run the face recognition algorithm during the automated testing process of the automated testing module, and obtain the test results; The database call module, the data preprocessing module, and the dataset management module cooperate to form the dataset source of the system. The database call module configures and batches downloads the dataset from the large-scale face test database for this detection, calls the test interface through the device interface debugging module and the device under test, and uses the device under test to collect the face images of the test personnel on site. The data preprocessing module preprocesses the face images collected on site by the device under test and uses them as the on-site collection dataset in the test database this time; Determine the quantity of the on-site collection dataset and the quantity of the dataset downloaded from the face test database according to the ratio rule. The dataset management module confirms the data scale according to the detection requirements of the current test. The database call module downloads the corresponding dataset from the test database by percentage, and manages the dataset formed by the face images collected on site prepared by the data preprocessing module, aggregates the downloaded dataset and the on-site collection dataset, and centrally processes them with the single-person dataset as the unit; The automated testing module is used in a computer or server environment to cooperate with the dataset management module to separately obtain the algorithm operation data of the device under test by calling different interface functions according to the test program through the device interface debugging module for the face images in the corresponding target set and probe set in the single-person dataset, and perform data processing on the obtained results to achieve the detection of performance indicators of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)"; carry out the first stage of the test: feature extraction. The automated testing module drives the device interface debugging module to initialize, calls the feature extraction interface function of the device under test, and performs feature extraction on each face image in the target set and the probe set in the test database managed by the dataset management module. The implementation of feature extraction is run by the face recognition algorithm of the device under test; the automated testing module analyzes and processes the face feature file library, the total number of samples, the number of successful extractions, and their corresponding identity identification information in the target set and the probe set respectively, checks the corresponding relationship between the two datasets, and records and stores them; Carry out the second stage of the test: feature comparison. The automated testing module drives the device interface call module to initialize, calls the feature comparison interface function of the device under test, and performs feature comparison on the feature data of the probe set and the feature data of the target set through the face recognition algorithm of the device under test. The automated testing module obtains the feature comparison results and analyzes and processes the similarity of each comparison with the similarity value accurate to the preset value and its corresponding data information; carry out the third stage of the test: calculation of the performance test results of "False Acceptance Rate (FAR) and False Rejection Rate (FRR)"; The quality evaluation module detects the performance indicators of face image quality evaluation for the on-site live face images collected by the device under test by calling the image algorithm module, and can simultaneously perform the same for the visualized face images of the certificates collected by the device under test.

2. The algorithm evaluation system for performance detection of human identity verification devices according to claim 1, wherein The evaluation system further includes a function module for hierarchical permission management of users, and the function module includes one or more of a user login module, a project registration module, and a user management module.

3. The algorithm evaluation system for performance detection of human identity verification devices according to claim 1, characterized in that The evaluation system includes an information statistics module for data analysis and processing of test results, and the information statistics module includes one or more of a test result module, a project management module, a project statistics module, and an algorithm statistics module.

4. The algorithm evaluation system for performance detection of human identity verification devices according to claim 1, wherein The automated test module in the system cooperates with the device interface debugging module, the data preprocessing module, the database call module, and the dataset management module to detect the "False Acceptance Rate (FAR) and False Rejection Rate (FRR)" performance of the device under test.

5. The algorithm evaluation system for performance detection of human identity verification devices according to claim 4, characterized in that The device interface debugging module obtains the face recognition algorithm test interface of the device under test; The data preprocessing module preprocesses the face images collected on-site by the device under test and uses them as the on-site collection database in this test database; The database call module calls and downloads the database required for this test; the dataset management module sets up a data security mechanism to manage the total test database for this test, and the automated test module calls the above modules to implement the "False Acceptance Rate (FAR) and False Rejection Rate (FRR)" performance detection.

6. The algorithm evaluation system for performance detection of human identity verification devices according to claim 1, wherein, The quality evaluation module cooperates with the device interface call module, the data preprocessing module, and the image algorithm module to evaluate the image quality of the face images collected on-site by the device under test.

7. The algorithm evaluation system for performance detection of human identity verification devices according to claim 6, wherein The device interface debugging module obtains and uploads the on-site live face images, ID photos, or visualized face images of certificates collected by the device under test; the data preprocessing module performs face detection tests on the images obtained by the device interface call module; the quality evaluation module evaluates the face image quality of the uploaded images by calling the image algorithm module, and the image algorithm module performs automated tests on the face images collected on-site and exported by the device under test according to the set technical requirements. The quality evaluation module analyzes and processes the data generated in cooperation with the image algorithm module and gives a compliance evaluation.

8. A detection method for detecting the performance of a human identity verification device, characterized in that, Including: Determine the quantity of the on-site collection dataset and the quantity of the dataset downloaded from the large-scale face test database according to the ratio rule, batch-download the datasets configured according to the rule from the large-scale face test database, and at the same time use the face images of the test personnel collected on-site by the device under test. Preprocess the face images collected on-site by the device under test and use them as the on-site collection database in this test database, and push them to the device under test in sequence according to the configuration rule for face algorithm operation; Control and obtain the test results of feature extraction and feature comparison of the device under test through the test function interface call, implement performance index detection, and carry out the first stage of testing: feature extraction. The automated test module drives the initialization of the device interface debugging module, calls the feature extraction interface function of the device under test, and performs feature extraction on each face image in the target set and the probe set in the test database managed by the dataset management module. The implementation of feature extraction is carried out by the face recognition algorithm of the device under test; the automated test module analyzes and processes the information such as the face feature file library, the total number of samples, the number of successful extractions, and their corresponding identity identifiers in the target set and the probe set respectively, checks the corresponding relationship between the two datasets and records and stores them. Carry out the second stage of testing: feature comparison. The automated test module drives the initialization of the device interface call module, calls the feature comparison interface function of the device under test, performs feature comparison on the feature data of the probe set and the feature data of the target set through the face recognition algorithm of the device under test, and the automated test module obtains the feature comparison result and analyzes and processes the similarity value and its corresponding data information for each comparison accurate to the preset value.

9. The detection method for detecting the performance of a human identity verification device according to claim 8, characterized in that, The detection method includes image quality performance detection of the face images collected by the device under test, including: Automatically test the live face images collected on site through image algorithms according to the set technical requirements, and then analyze and process through the quality evaluation module to obtain the image quality evaluation result; Automatically test the visible face images of the certificates collected through image algorithms according to the set technical requirements, and then analyze and process through the quality evaluation module to obtain the image quality evaluation result.

10. The detection method for performance detection of a human identity verification device according to claim 8, characterized in that, In the detection method, the test database is downloaded according to the data security mechanism, and is pushed to the device under test in sequence according to the configuration rules for face algorithm operation, and the face image data is encrypted and desensitized for use with reference to the mapping relationship; it is ensured that the face data in the called face test database cannot be exported and reused in the device under test, thereby ensuring the safe use of the face data in the face test database.

11. The detection method for performance detection of human identity verification devices according to claim 8, characterized in that, The detection method includes the specific detection steps for testing the performance indicators of the false acceptance rate (FAR) and the false rejection rate (FRR) of the human-machine verification device.

Images