High-frequency access personnel statistical method and device based on face recognition

Abstract

The invention discloses a high-frequency access personnel statistical method and device based on face recognition, and belongs to the technical field of information security. The method comprises the following steps: extracting face features of a face image of which the image quality score exceeds a first preset threshold value; when the periodic update time reaches or reaches a trigger condition, acquiring all face features without face identifiers from a Faiss database; sorting all the face features to which the face identifiers are not distributed based on the timestamps; performing judgment and database updating operation on each sorted face feature; obtaining a to-be-recognized face feature, and querying whether a face feature whose similarity with the to-be-recognized face feature is greater than a third preset threshold exists in a Faiss database; and performing high-frequency access personnel judgment based on a judgment result. According to the invention, accurate recognition and matching of face features are realized. The method can be widely applied to multiple fields such as safety monitoring, commercial management and public service.

Description

Technical Field

[0001] The present application belongs to the field of information security technology, and in particular, relates to a method and device for counting high-frequency in-and-out personnel based on face recognition. Background Art

[0002] At present, there are few methods for counting high-frequency in-and-out personnel based on face recognition. There are some similar passenger flow calculation methods. Among them, the vision-based methods include single-lens face recognition and gate face recognition. The former has the disadvantage of a small monitoring range, and the latter has the disadvantage of clumsy gates and large size. When the passenger flow is large, it is easy to cause long queues, and the human body posture requirements are high, and few faces are captured at the same time. There are also some based on infrared cameras or antennas, which have the disadvantages of difficult installation and high cost. The general face recognition method is to first prepare high-definition frontal face pictures into the database, and then search and compare. Summary of the invention

[0003] The present application provides a method and device for counting high-frequency in-and-out personnel based on face recognition, which are used to solve the above problems or at least partially solve the above problems.

[0004] In a first aspect, a method for counting high-frequency in-and-out personnel based on face recognition is disclosed, the method comprising:

[0005] Step S1: multiple image acquisition devices work in parallel to acquire multiple images; perform human body detection on each image to determine a number of human portraits; for each human portrait, cut off the upper part of the human body and extract the face image; extract the face features of the face image whose image quality score exceeds a first preset threshold; store each face feature in the Faiss database, and store the face image corresponding to the face feature and the timestamp of acquiring the face in the MySQL database;

[0006] Step S2: when the periodic update time arrives or the trigger condition is met, the facial features of all unassigned facial identifiers are obtained from the Faiss database; and the facial features of all unassigned facial identifiers are sorted based on the timestamps corresponding to the facial features;

[0007] For each sorted facial feature, the following operations are performed:

[0008] Step S21: determining the image quality score corresponding to the facial feature; if the image quality score exceeds a second threshold, the facial feature is used as a facial feature to be processed, and the process proceeds to step S22; otherwise, the facial feature is processed;

[0009] Step S22: comparing the facial features to be processed with the facial features stored in the Faiss database, and determining whether the facial features to be processed are new facial features based on the comparison results;

[0010] If it is a newly appeared facial feature, a facial identifier is assigned to the facial feature to be processed; the facial identifier corresponding to the facial feature to be processed is completed in the MySQL database; the facial identifier of the facial feature to be processed is stored in the Redis database, the appearance frequency of the facial identifier is recorded as 1, and the expiration time corresponding to the facial identifier is set;

[0011] Otherwise, the face identifier corresponding to the face feature to be processed is completed in the MySQL database; a record of the face identifier is added in the Redis database, the face identifier of the face feature to be processed is recorded in the newly added record, and the occurrence frequency corresponding to the face identifier is updated; the face identifier corresponding to the face feature to be processed is completed in the Faiss database;

[0012] When the facial identifier stored in the Redis database reaches its corresponding expiration time and no corresponding record is added, all records of the facial identifier are deleted, and the occurrence frequency corresponding to the facial identifier is cleared to zero, and the facial identifier is released; when the Redis database deletes the facial identifier, the MySQL database and the Faiss database delete all the contents corresponding to the facial identifier and clear them to zero, and the facial identifier is released;

[0013] Step S3: obtaining facial features to be identified, and querying in the Faiss database whether there are facial features whose similarity with the facial features to be identified is greater than a third preset threshold;

[0014] If so, obtaining the appearance frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than the third preset threshold, and when the appearance frequency exceeds the fourth preset threshold, determining that the person corresponding to the face feature to be identified is a high-frequency person;

[0015] If not, the facial features to be identified are stored in the Faiss database to meet the triggering condition.

[0016] Preferably, the step S1 comprises:

[0017] Step S11: reading the stream addresses of multiple image acquisition devices through OpenCV, and obtaining unprocessed images from each image acquisition device;

[0018] Step S12: Perform human body detection on each image using YOLOv5 to determine a number of portraits; for each portrait, cut off the upper half of the human body and extract the face image, and the face image is extracted in the following manner: determine that the length of the upper half of the human body is h1 and the width is w; cut off a part of size w*w from the upper left corner of the upper half of the human body as the face image or cut off a part of size w*w from the upper half of the human body from top to bottom. The part is taken as the face image;

[0019] Step S13: extracting facial features of facial images whose image quality scores exceed a first preset threshold using a resnet50 network model, and normalizing all facial features; the image quality score is determined based on the SDD-FIQA score, illumination, facial posture, and facial clarity;

[0020] Step S14: storing the normalized facial features and the timestamp of obtaining the portrait corresponding to the facial features in a MySQL database, wherein the timestamp is the time when the image acquisition device obtains the portrait.

[0021] Preferably, in step S2, the face identifiers are allocated in ascending order according to currently available identifiers.

[0022] Preferably, the step S3, obtaining the occurrence frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than a third preset threshold, includes:

[0023] The Redis database, the MySQL database, and the Faiss database add and / or modify the relevant information corresponding to the face identifier corresponding to the face feature whose similarity to the face feature to be identified is greater than a third preset threshold;

[0024] Determine the first timestamp when the facial feature to be identified is obtained and the facial identifier corresponding to the facial feature to be identified. If the interval time between other timestamps corresponding to the facial identifier stored in the MySQL database and the first timestamp is less than a second preset threshold, the frequency corresponding to the facial identifier remains unchanged; otherwise, the frequency corresponding to the facial identifier is increased by 1.

[0025] Preferably, a whitelist is set in the Redis database, and if the person who frequently enters and exits is a person on the whitelist, no alarm information is generated; otherwise, an alarm information is generated.

[0026] In a second aspect, a high-frequency entry and exit personnel counting device based on face recognition is disclosed, the device comprising:

[0027] Acquisition module: configured as multiple image acquisition devices working in parallel to acquire multiple images; perform human body detection on each image to determine a number of portraits; for each portrait, cut off the upper part of the human body and extract the face image; extract the face features of the face image whose image quality score exceeds a first preset threshold; store each face feature in the Faiss database, and store the face image corresponding to the face feature and the timestamp of acquiring the portrait in the MySQL database;

[0028] Sorting module: configured to obtain all facial features that are not assigned with facial identifiers from the Faiss database when the periodic update time arrives or the trigger condition is met; sort all facial features that are not assigned with facial identifiers based on the timestamps corresponding to the facial features;

[0029] For each sorted facial feature, the following submodules are triggered:

[0030] The first submodule is configured to determine the image quality score corresponding to the facial feature, and if the image quality score exceeds the second threshold, the facial feature is used as a facial feature to be processed, triggering the second submodule; otherwise, the facial feature is processed;

[0031] The second submodule is configured to compare the facial features to be processed with the facial features stored in the Faiss database, and determine whether the facial features to be processed are newly appeared facial features based on the comparison results;

[0032] If it is a newly appeared facial feature, a facial identifier is assigned to the facial feature to be processed; the facial identifier corresponding to the facial feature to be processed is completed in the MySQL database; the facial identifier of the facial feature to be processed is stored in the Redis database, the appearance frequency of the facial identifier is recorded as 1, and the expiration time corresponding to the facial identifier is set;

[0033] Otherwise, the face identifier corresponding to the face feature to be processed is completed in the MySQL database; a record of the face identifier is added in the Redis database, the face identifier of the face feature to be processed is recorded in the newly added record, and the occurrence frequency corresponding to the face identifier is updated; the face identifier corresponding to the face feature to be processed is completed in the Faiss database;

[0034] When the facial identifier stored in the Redis database reaches its corresponding expiration time and no corresponding record is added, all records of the facial identifier are deleted, and the occurrence frequency corresponding to the facial identifier is cleared to zero, and the facial identifier is released; when the Redis database deletes the facial identifier, the MySQL database and the Faiss database delete all the contents corresponding to the facial identifier and clear them to zero, and the facial identifier is released;

[0035] Recognition module: configured to obtain facial features to be recognized, and query in the Faiss database whether there is a facial feature whose similarity with the facial features to be recognized is greater than a third preset threshold;

[0036] If so, obtaining the appearance frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than the third preset threshold, and when the appearance frequency exceeds the fourth preset threshold, determining that the person corresponding to the face feature to be identified is a high-frequency person;

[0037] If not, the facial features to be identified are stored in the Faiss database to meet the triggering condition.

[0038] In a third aspect, an electronic device is disclosed, the electronic device comprising:

[0039] at least one processor; and

[0040] a memory communicatively connected to the at least one processor; wherein,

[0041] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described above.

[0042] In a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is disclosed, wherein the computer instructions are used to cause the computer to execute the method as described above.

[0043] This application has the following technical effects:

[0044] This application can dynamically enter data across cameras in multiple areas, and update the face database while detecting and identifying. You can use the surveillance cameras owned by the community or shopping mall to perform face recognition and assign face IDs (short for "Identifier", all with ID). Use the Redis database to store the face IDs detected in multiple videos, and use high-frequency entry and exit personnel to count the high-frequency entry and exit personnel of the entire floor, the entire building, or even the entire community. When high-frequency personnel appear, an alarm can be issued, and the portraits and face screenshots of high-frequency personnel are stored in the MySQL database for subsequent tracking and retrieval. Not only can it overcome the above shortcomings, but it can also further perform high-frequency face statistics and alarms based on passenger flow statistics. In the security field, it helps security managers to promptly discover abnormal behavior or potential safety hazards; in the commercial field, it assists merchants to understand the frequency of customer visits, thereby optimizing product layout; in the public service field, it assists managers to understand the use of venues and optimize resource allocation.

[0045] This application uses deep learning algorithms to achieve accurate recognition and matching of facial features. It can be widely used in security monitoring, business management, public services and other fields. This application records and analyzes the entry and exit of people in a specific area in real time, provides important data support for managers, assists managers in security monitoring and risk assessment, optimizes resource allocation, and improves management efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] Figure 1 The figure is a flowchart of a method for counting high-frequency in-and-out personnel based on face recognition;

[0047] Figure 2 The schematic diagram of the architecture of the high-frequency personnel counting method based on face recognition;

[0048] Figure 3 Schematic diagram of target detection and feature extraction process;

[0049] Figure 4 This is a schematic diagram of the dynamic face storage process;

[0050] Figure 5 This is a schematic diagram of the process of counting high-frequency in-and-out personnel;

[0051] Figure 6 This is a schematic diagram of the high-frequency personnel information query system architecture;

[0052] Figure 7 The figure is a schematic diagram of the structure of a high-frequency personnel counting device based on face recognition. DETAILED DESCRIPTION

[0053] The embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0054] like Figure 1-Figure 2 As shown, the present application provides a method for counting high-frequency in-and-out personnel based on face recognition, the method comprising:

[0055] Step S1: multiple image acquisition devices work in parallel to acquire multiple images; perform human body detection on each image to determine a number of human portraits; for each human portrait, cut off the upper part of the human body and extract the face image; extract the face features of the face image whose image quality score exceeds a first preset threshold; store each face feature in the Faiss database, and store the face image corresponding to the face feature and the timestamp of acquiring the face in the MySQL database;

[0056] Step S2: when the periodic update time arrives or the trigger condition is met, the facial features of all unassigned facial identifiers are obtained from the Faiss database; and the facial features of all unassigned facial identifiers are sorted based on the timestamps corresponding to the facial features;

[0057] For each sorted facial feature, the following operations are performed:

[0058] Step S21: determining the image quality score corresponding to the facial feature; if the image quality score exceeds a second threshold, the facial feature is used as a facial feature to be processed, and the process proceeds to step S22; otherwise, the facial feature is processed;

[0059] Step S22: comparing the facial features to be processed with the facial features stored in the Faiss database, and determining whether the facial features to be processed are new facial features based on the comparison results;

[0060] If it is a newly appeared facial feature, a facial identifier is assigned to the facial feature to be processed; the facial identifier corresponding to the facial feature to be processed is completed in the MySQL database; the facial identifier of the facial feature to be processed is stored in the Redis database, the appearance frequency of the facial identifier is recorded as 1, and the expiration time corresponding to the facial identifier is set;

[0061] Otherwise, the face identifier corresponding to the face feature to be processed is completed in the MySQL database; a record of the face identifier is added in the Redis database, the face identifier of the face feature to be processed is recorded in the newly added record, and the occurrence frequency corresponding to the face identifier is updated; the face identifier corresponding to the face feature to be processed is completed in the Faiss database;

[0062] When the facial identifier stored in the Redis database reaches its corresponding expiration time and no corresponding record is added, all records of the facial identifier are deleted, and the occurrence frequency corresponding to the facial identifier is cleared to zero, and the facial identifier is released; when the Redis database deletes the facial identifier, the MySQL database and the Faiss database delete all the contents corresponding to the facial identifier and clear them to zero, and the facial identifier is released;

[0063] Step S3: obtaining facial features to be identified, and querying in the Faiss database whether there are facial features whose similarity with the facial features to be identified is greater than a third preset threshold;

[0064] If so, obtaining the appearance frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than the third preset threshold, and when the appearance frequency exceeds the fourth preset threshold, determining that the person corresponding to the face feature to be identified is a high-frequency person;

[0065] If not, the facial features to be identified are stored in the Faiss database to meet the triggering condition.

[0066] The step S1 comprises:

[0067] Step S11: reading the stream addresses of multiple image acquisition devices through OpenCV, and obtaining unprocessed images from each image acquisition device;

[0068] Step S12: Perform human body detection on each image using YOLOv5 to determine a number of portraits; for each portrait, cut off the upper half of the human body and extract the face image, and the face image is extracted in the following manner: determine that the length of the upper half of the human body is h1 and the width is w; cut off a part of size w*w from the upper left corner of the upper half of the human body as the face image or cut off a part of size w*w from the upper half of the human body from top to bottom. The part is taken as the face image;

[0069] Step S13: extracting facial features of facial images whose image quality scores exceed a first preset threshold using a resnet50 network model, and normalizing all facial features; the image quality score is determined based on the SDD-FIQA score, illumination, facial posture, and facial clarity;

[0070] Step S14: storing the normalized facial features and the timestamp of obtaining the portrait corresponding to the facial features in a MySQL database, wherein the timestamp is the time when the image acquisition device obtains the portrait.

[0071] like Figure 3 As shown, the present application provides specific embodiments of target detection and feature extraction.

[0072] Step 101, obtaining the stream address of the camera and extracting the latest real-time image. By reading multiple camera stream addresses through OpenCV, each camera will get a cached image queue. Each time the latest image in the queue is obtained, it can ensure that there is no delay in target detection and recognition when multiple channels are parallel.

[0073] Step 102, human body detection and human body screenshot. The human body detection algorithm here uses the general yolov5 target detection algorithm. An innovative point of this application is that the entire pedestrian picture is not intercepted, because the required face is generally in the upper half of the human body. When standing, the face accounts for a small proportion, and when squatting, it accounts for a large proportion, but it is in the upper half (public places, not considering extreme inverted pedestrians). This application intercepts the upper half of the human body, which usually contains the face. If the human body is h high and w wide, strategy one is to intercept a w*w picture from the upper left corner, and strategy two is to intercept a h / 3*w part. Strategy two will make face detection faster but the probability of intercepting incomplete faces is greater. The purpose of interception is to reduce the picture and speed up the detection speed of subsequent face detection. Assume that the human body screenshots detected by cameras 1, 2 and 3 are body1, body2, and body3 respectively.

[0074] Step 103, face detection and face screenshot. After successfully detecting the human body and capturing the corresponding area, face detection is performed next. Face detection uses the Mtcnn face detection algorithm or the Retinaface face detection algorithm to perform face detection on the human body screenshot of step 102 in parallel and capture the face frame to obtain face1, face2 and face3.

[0075] Step 104, face quality evaluation and face image screening. In order to achieve higher face recognition accuracy and more accurate calculation of high-frequency personnel, it is necessary to remove faces that do not meet the quality standards. There are two methods for face quality evaluation: supervised and unsupervised. This application combines the two solutions. The unsupervised SDD-FIQA (Similarity Distribution Distance for Face Image Quality Assessment, an unsupervised quality assessment method based on face similarity distribution distance) face quality assessment method gives the overall quality score of the face, and combines the supervised training illumination score, face posture score and face clarity score model to comprehensively evaluate the face quality. Assuming that the SDD-FIQA quality score is q1, the illumination score is q2, the face posture score is q3, and the face clarity score is q4, the comprehensive face quality score score = A*q1+B*q2+C*q3+(1-ABC)*q4, where A, B, and C are adjustable weight parameters. Assuming that the scores of face1, face2 and face3 are all above 40 points (adjustable threshold), face1, face2 and face3 are retained at this stage. The face scores are recorded and saved here because the quality requirements of the faces to be stored later and the faces used for comparison are different.

[0076] Step 105, facial feature extraction. Use the resnet50 network model to extract facial features and normalize them. The features corresponding to face1, face2, and face3 are x1, x2, and x3, respectively. At this time, the video recording and pedestrian detection timestamps, face detection timestamps, and feature extraction timestamps are recorded and saved to trace back the images of people who frequently enter and exit.

[0077] In the present invention, the Faiss database, the Redis database, and the MySQL database together form a dynamic face database. Faiss mainly stores face features for face search, and the Redis database mainly stores face IDs and update expiration times. If the quality score of the face to be identified exceeds the threshold and the feature comparison in Faiss does not exceed the threshold, a new ID and expiration time are assigned to the face to be identified in the Redis database, and each time this face appears, the face portrait and appearance timestamp and other information are stored in MySQL.

[0078] If the face does not appear within the expiration time, delete the face ID and expiration time record in the Redis database, delete the face feature vector in the Faiss database, and delete the corresponding face portrait, ID, timestamp and other information in the MySQL database. Optionally, the MySQL database is relatively independent and can be kept for a monthly rotation. If no face is detected for a long time, all three databases will be cleared.

[0079] Furthermore, in step S2, the face identifiers are allocated in ascending order according to currently available identifiers.

[0080] like Figure 4 As shown, this application provides a specific embodiment of dynamic face storage. It includes the following steps:

[0081] Step 201, face feature sorting unit. The target detection and feature extraction module obtains face1, face2 and face3 and their corresponding features x1, x2, x3, and sorts face1, face2 and face3 in the order of obtaining features x1, x2, x3. If x2, x1, x3 are extracted in the order of parallel recognition, the order of face acquisition is face2, face1, face3.

[0082] Step 202, faces are stored in the database one by one, and face2 is considered first. Step 104: After the faces have passed the quality evaluation model, faces with a quality score lower than 40 are directly discarded (faces with extremely poor filtering quality that affect comparison, the threshold is adjustable). Here, they are compared with the face storage threshold of 70 (the threshold is adjustable). If the threshold is exceeded, the feature x2 of face2 can be added to the face database used for subsequent face searches. (For the convenience of drawing and explanation, face1 and face3 will pass this threshold later). It is assumed here that the face database is empty, and the ID of face2 is assigned to 0. The ID assignment is dynamic. If face2 does not appear for a long time, its ID is removed (described in steps 214-216). If 0 is occupied when it appears next time, the first minimum free ID is taken. For example, the IDs in the database are 0, 1, 2, 4, 5 and 9. At this time, if face2 appears again, its ID is assigned to 3. The face ID of the face storage method proposed in this application can be stored (added), can be removed (ID is deleted when timeout), and the ID can also be changed. This application automatically and dynamically updates the face database, which keeps the database simple while keeping the ID simple and coherent. The features and IDs in the database are stored in Faiss (Facebook's open source AI similarity search tool), which can support millisecond-level face searches in a database of millions of faces.

[0083] Step 203: Faces are stored in the database one after another, and then face1 is considered to be stored in the database. Search in the Faiss face database, that is, perform feature comparison with face2 in the database.

[0084] Step 204: Determine whether the comparison score exceeds a threshold. If the comparison score of face1 and face2 exceeds the threshold, proceed to step 206; otherwise, proceed to step 205.

[0085] Step 205: add face1 to the database. If the similarity score of face1 and face2 in step 204 is lower than the threshold, face1 and face2 are not similar, and face1 is added to the database. At this time, there is only ID 0 in the database, and the first minimum free ID is 1. The ID assigned to face1 is 1.

[0086] Step 206, processing of face1. If the similarity score of face1 and face2 in step 204 is higher than the threshold, face1 and face2 are the same person, and face1 is not assigned a new ID, but the relevant information of face1 is saved in the MySQL database, including the time when face1 appeared, the screenshot of the human body to which face1 belongs, the ID of face1, etc., to facilitate subsequent search and backtracking.

[0087] Step 207: Continue to consider adding face3 to the database. Search and compare face3 in the Faiss face database, that is, perform feature comparison with face1 and face2.

[0088] Step 208: Determine whether the comparison score exceeds a threshold. If the comparison score of face3 and face1 or face2 exceeds the threshold, proceed to step 209; otherwise, proceed to step 210.

[0089] Step 209, processing of face3. If the similarity score of face3 and face1 or face2 is higher than the threshold, face3 and face or face2 are the same person, and face3 is not assigned a new ID, but face3 is saved in the MySQL database together with the relevant information of face1 and face2. At this time, the Faiss face database contains the facial features and IDs of face1 and face2.

[0090] Step 210, face3 is added to the database. If the comparison scores of face3 and face1 or face2 are all lower than the threshold, then face3 and face or face2 are not the same person. At this time, there are IDs 0 and 1 in the database, and face3's ID is assigned to 2. At this time, the Faiss face database contains the facial features and IDs of face1, face2, and face3. At the same time, the relevant information of face1, face2, and face3 is saved in the MySQL database.

[0091] Step 211, feature comparison of face3 and face2. Step 206 finds that face1 and face2 are the same person, and then obtains the similarity between face3 and face2.

[0092] Step 212: Determine whether the comparison score exceeds a threshold. If the comparison score of face3 and face1 exceeds the threshold, proceed to step 214; otherwise, proceed to step 213.

[0093] Step 213, face3 is added to the database. At this time, there is only ID 0 in the database, and face3's ID is assigned to 1. At this time, the Faiss face database contains the facial features and IDs of face2 and face3. At the same time, the relevant information of face1, face2 and face3 is saved in the MySQL database.

[0094] Step 214, processing of face3. If the similarity score of face3 and face1 or face2 is higher than the threshold, face3 and face or face2 are the same person, and face3 is not assigned a new ID, but face3 is saved in the MySQL database together with the relevant information of face1 and face2. At this time, the Faiss face database contains the facial features and ID of face2.

[0095] Step 215: Determine whether the algorithm runtime is an integer multiple of 10 minutes. Determine whether the algorithm runtime is an integer multiple of 10*60s. If yes, proceed to step 216; otherwise, proceed to step 217.

[0096] Step 216: Store the facial features and ID locally, and read the historical facial data when restarting the algorithm. This can prevent the facial information from being lost when the system is restarted.

[0097] Step 217, once the storage is completed, return to the target detection and feature extraction module to continue execution.

[0098] The step S3, obtaining the frequency corresponding to the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than a third preset threshold, includes:

[0099] The Redis database, the MySQL database, and the Faiss database add and / or modify the relevant information corresponding to the face identifier corresponding to the face feature whose similarity to the face feature to be identified is greater than a third preset threshold;

[0100] Determine the first timestamp when the facial feature to be identified is obtained and the facial identifier corresponding to the facial feature to be identified. If the interval time between other timestamps corresponding to the facial identifier stored in the MySQL database and the first timestamp is less than a second preset threshold, the frequency corresponding to the facial identifier remains unchanged; otherwise, the frequency corresponding to the facial identifier is increased by 1.

[0101] Furthermore, a whitelist is set in the Redis database. If the person who frequently enters and exits is a person on the whitelist, no alarm information is generated; otherwise, an alarm information is generated.

[0102] Furthermore, the MySQL database also stores facial identifications, facial screenshots, and persons to whom the faces belong of people who frequently enter and exit.

[0103] like Figure 5 As shown, the present invention provides a specific embodiment of high-frequency entry and exit personnel statistics, including:

[0104] Step 301, count the number of times the current face appears. In the previous dynamic face storage module, each person's ID and frequency of appearance 1 will be stored in the Redis database when it first appears. Each time the face appears again and meets the conditions, the frequency will be increased by 1. Assuming that face1 appears n times and meets the conditions at this time, the frequency is n. At the same time, an expiration time is set for each appearance, that is, the interval time of 5 days (configurable) mentioned in the following step 308. If face1 does not appear again for more than 5 days after the last appearance, the face ID and frequency of face1 in Redis will be automatically cleared.

[0105] Step 302: Search the face database for faces that have passed the face quality score. If the new face passes the face quality evaluation unit in the target detection and feature extraction module, a face search is performed in the Faiss database.

[0106] Step 303: Determine whether the search is successful. If a similar face is found in the database, proceed to step 306; otherwise, proceed to step 304.

[0107] Step 304: determine whether the face quality score exceeds the storage threshold. If it exceeds the threshold, proceed to step 305. Otherwise, you can ignore or record low-quality faces, which we ignore here.

[0108] Step 305: Add new face to the database. Return to the dynamic face storage module to continue execution. The new face ID is the smallest free ID in the database. Assign the smallest free ID in the database to the new face, and save the new face screenshot, timestamp and other information to the high-frequency personnel information query system. Then, continue to return to the target detection and feature extraction module.

[0109] Step 306: Search result judgment: If the similarity between the searched face and face1 exceeds the threshold, the person appearance frequency is calculated based on the frequency of face1; otherwise, the frequency is calculated based on the frequency of other corresponding persons searched.

[0110] Step 307: Calculate the time interval between the last two occurrences using the expiration time of Redis.

[0111] Step 308: Determine whether the time interval exceeds a threshold value (eg, 5 days). If yes, proceed to step 309; otherwise, proceed to step 310.

[0112] Step 309: If face1 does not appear for a long time, delete its information. Delete the relevant information of face1 from Redis, Faiss and MySQL, including ID, frequency, facial features, screenshots and timestamps. If you want to save the person for a longer time, you can increase the interval time threshold mentioned in step 308, for example, set it to 3 months, but you have to consider the limitation of computer hard disk storage.

[0113] Step 310: Continue to determine whether the time interval is less than a threshold value (eg, 1 minute). If yes, proceed to step 311; otherwise, proceed to step 312.

[0114] Step 311: Repeated appearance in a short time does not increase the frequency. Since the time interval between the two appearances is short, it may be adjacent frames where a person is standing or walking, and the frequency of face1's entry and exit is not increased, but the face information is saved in MySQL for backtracking query.

[0115] Step 312: Increase the frequency of personnel. If the time interval is long enough, increase the frequency of face1 by 1. Step 313: Determine whether the updated frequency is greater than a threshold (such as 20 times). If yes, proceed to step 314; otherwise, proceed to step 316.

[0116] Step 314: Determine the frequent entry and exit personnel. Mark face1 as a frequent entry and exit personnel, and store its information in the MySQL database.

[0117] Step 315: System notification alarm and query. Optional functions include system alarm and whitelist settings. For people who frequently enter and exit, the system can send alarm notifications; at the same time, a whitelist can be set to exclude frequency statistics of specific people (such as managers or cleaning staff).

[0118] Step 316: Continue to wait for the appearance of new persons. Regardless of whether face1 is considered as a high-frequency person, the process will return to the target detection and feature extraction module to wait for the appearance of new faces and perform the next round of statistics.

[0119] This application is based on a high-frequency entry and exit personnel counting method for face recognition. It uses the video streams of multiple cameras in public places. First, through the face detection and feature extraction module: the multi-process detection method is used to detect pedestrians and their corresponding faces, and the face pictures are intercepted and the faces with good enough quality are screened to extract face features. Then, through the set dynamic face storage module: this module stores the feature data output by the face feature extraction module in the Faiss database, and assigns a unique ID identifier and expiration time to the newly appeared face and stores it in the Redis database. Finally, through the high-frequency entry and exit personnel statistics module: the frequency of each person in each video channel is counted and the face screenshot and ID of the person are recorded when the frequency exceeds the threshold. These three modules work closely together and give full play to the advantages of multi-channel video. They can effectively track and identify people who repeatedly enter and exit, wander or stay for a long time, and automatically alarm and record relevant information when high-frequency personnel are found, which provides great convenience for management personnel and helps to timely discover and deal with lingering personnel or suspicious personnel.

[0120] The present application provides a method and system for counting high-frequency in-and-out personnel based on face recognition.

[0121] The high-frequency personnel statistics system includes two parts: the intelligent recognition system 20 and the high-frequency personnel information query system 21. The intelligent recognition system 20 is responsible for detecting human faces and dynamically storing human faces in the database, and calculating the frequency of people entering and leaving the database by searching and matching human faces in the database. Then, the information in the system 20 is stored in the high-frequency personnel information query system 21, so that managers can verify high-frequency personnel and further determine whether they are people staying in the community, suspicious people at the railway station, or customers with high purchasing intentions in the mall. The high-frequency personnel information query system 21 also provides a whitelist function. For managers or community cleaning staff who do not need to be alarmed, their IDs can be added to the whitelist, and the frequency of entry and exit of whitelisted personnel will not be counted to reduce unnecessary frequent alarms.

[0122] More specifically, the intelligent recognition system 20 includes three modules: a target detection and feature extraction module 201, a dynamic face storage module 202 and a high-frequency entry and exit personnel statistics module 203. Module 201 obtains human body screenshots, face images, face features, timestamps and other information from the corresponding camera through face detection, face quality evaluation and face recognition technologies. Module 202 compares and deduplicates the faces obtained by module 201 and dynamically stores them in the database. Module 203 calculates the frequency of appearance of people in the database through frequency calculation logic to calculate the high-frequency entry and exit personnel.

[0123] The framework diagram of the high-frequency personnel information query system shows the high-frequency personnel statistics method based on face recognition. The system framework contains three databases: Redis database, Faiss database and MySQL database. After target detection and feature extraction, the face information is stored in the Redis database and Faiss database through the dynamic face storage process. The calculation of high-frequency personnel depends on the data in the Redis database and Faiss database. The results of the high-frequency personnel statistics, together with other relevant information of target detection, are stored in the MySQL database. Users can query the high-frequency personnel information in the MySQL database. They can query by face ID, query the high-frequency personnel related to a camera through the camera, and query the high-frequency personnel appearing in a certain time period by specifying the timestamp interval.

[0124] This application proposes a method for counting high-frequency in-and-out personnel based on face recognition. It obtains face-related information based on target detection and feature extraction, proposes a dynamic face storage method and a high-frequency in-and-out personnel calculation method, and automatically maintains a face feature library and a high-frequency in-and-out personnel database that can be automatically increased, decreased and modified. High-frequency in-and-out personnel statistics can help security managers to promptly detect abnormal behavior or potential safety hazards in the security field; in the commercial field, it can assist merchants to understand the frequency of customer visits, thereby optimizing the layout of goods; in the public service field, it can assist managers to understand the use of venues and optimize resource allocation.

[0125] The solution proposed in this application incorporates several innovative points:

[0126] This application proposes a dynamic face storage method, which automatically and dynamically updates the face database, and allocates the first minimum free ID each time when entering the database, so that the database is kept simple while the ID is kept simple and coherent. And the expiration time function of Redis is used to automatically clear the face information that has not appeared for a long time, effectively managing the size of the database and storage efficiency.

[0127] This application proposes an efficient face recognition and high-frequency personnel counting system that integrates multiple databases, and innovatively builds a high-frequency personnel counting system that integrates three databases: Redis, Faiss, and MySQL. This integration method enables the system to utilize the cache characteristics of Redis for temporary data storage and fast access, utilize the efficient search capabilities of Faiss for facial feature matching, and utilize the persistent storage and complex query capabilities of MySQL to save statistical results and provide query services. This multi-database collaboration model greatly improves the overall performance and flexibility of the system. Among them, the features and IDs entered into the database are stored in Faiss (Facebook's open source AI similarity search tool), which can support millisecond-level face searches in a database of millions of faces, and is hundreds of times faster than the traditional circular comparison method when the face database is large.

[0128] This application performs frequency statistics based on the face ID stored in Redis, automatically identifies people who frequently enter and exit, and realizes real-time monitoring of abnormal behavior.

[0129] This application does not need to rely on a pre-built high-definition face library, and realizes dynamic face recognition across cameras in multiple monitoring areas and real-time updating of the same face library, greatly improving the flexibility and real-time performance of the system.

[0130] This application does not require pre-setting of a fixed face database and is suitable for various complex environments, such as communities, shopping malls, office buildings, etc., reducing the workload of preliminary preparation and maintenance costs.

[0131] This application utilizes existing surveillance camera resources for face recognition, without the need to install additional dedicated gates or infrared equipment, reducing hardware costs and installation difficulty.

[0132] The target detection and feature extraction modules in this application utilize classic open source algorithms, but this application has made some innovative optimizations in application, including obtaining the latest pictures in the opencv image queue, capturing half-body pictures containing faces, and a comprehensive evaluation scheme for face quality evaluation, which can realize real-time detection of target detection and recognition and improve the accuracy of face quality evaluation.

[0133] This application proposes a high-precision face recognition and similarity judgment solution, which improves the accuracy of face search by setting two thresholds: a recognizable face threshold and a database-entry face threshold.

[0134] This application provides flexible query and alarm functions. The system supports flexible query functions based on face ID and timestamp interval, allowing users to quickly obtain information about high-frequency in-and-out personnel as needed. At the same time, the system also supports optional alarm functions, which can automatically send alarm notifications according to user-set conditions, improving the practicality of the system and user experience.

[0135] This application has modular design and scalability: The system adopts a modular design strategy to decompose a complex system into multiple independent and functionally clear modules. This design not only improves the scalability and maintainability of the system, but also enables the system to be flexibly configured and expanded according to different application scenarios, meeting diverse needs.

[0136] like Figure 7 As shown, the present application provides a high-frequency entry and exit personnel counting device based on face recognition, the device comprising:

[0137] Acquisition module: configured as multiple image acquisition devices working in parallel to acquire multiple images; perform human body detection on each image to determine a number of portraits; for each portrait, cut off the upper part of the human body and extract the face image; extract the face features of the face image whose image quality score exceeds a first preset threshold; store each face feature in the Faiss database, and store the face image corresponding to the face feature and the timestamp of acquiring the portrait in the MySQL database;

[0138] Sorting module: configured to obtain all facial features that are not assigned with facial identifiers from the Faiss database when the periodic update time arrives or the trigger condition is met; sort all facial features that are not assigned with facial identifiers based on the timestamps corresponding to the facial features;

[0139] For each sorted facial feature, the following submodules are triggered:

[0140] The first submodule is configured to determine the image quality score corresponding to the facial feature, and if the image quality score exceeds the second threshold, the facial feature is used as a facial feature to be processed, triggering the second submodule; otherwise, the facial feature is processed;

[0141] The second submodule is configured to compare the facial features to be processed with the facial features stored in the Faiss database, and determine whether the facial features to be processed are newly appeared facial features based on the comparison results;

[0142] If it is a newly appeared facial feature, a facial identifier is assigned to the facial feature to be processed; the facial identifier corresponding to the facial feature to be processed is completed in the MySQL database; the facial identifier of the facial feature to be processed is stored in the Redis database, the appearance frequency of the facial identifier is recorded as 1, and the expiration time corresponding to the facial identifier is set;

[0143] Otherwise, the face identifier corresponding to the face feature to be processed is completed in the MySQL database; a record of the face identifier is added in the Redis database, the face identifier of the face feature to be processed is recorded in the newly added record, and the occurrence frequency corresponding to the face identifier is updated; the face identifier corresponding to the face feature to be processed is completed in the Faiss database;

[0144] When the facial identifier stored in the Redis database reaches its corresponding expiration time and no corresponding record is added, all records of the facial identifier are deleted, and the occurrence frequency corresponding to the facial identifier is cleared to zero, and the facial identifier is released; when the Redis database deletes the facial identifier, the MySQL database and the Faiss database delete all the contents corresponding to the facial identifier and clear them to zero, and the facial identifier is released;

[0145] Recognition module: configured to obtain facial features to be recognized, and query in the Faiss database whether there is a facial feature whose similarity with the facial features to be recognized is greater than a third preset threshold;

[0146] If so, obtaining the appearance frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than the third preset threshold, and when the appearance frequency exceeds the fourth preset threshold, determining that the person corresponding to the face feature to be identified is a high-frequency person;

[0147] If not, the facial features to be identified are stored in the Faiss database to meet the triggering condition.

[0148] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application is described in detail with reference to the above embodiments, a person skilled in the art should understand that the technical solutions described in the above embodiments can still be modified, or some or all of the technical features can be replaced by equivalents, and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for counting high-frequency in-and-out personnel based on face recognition, characterized in that: The method comprises the following steps: Step S1: multiple image acquisition devices work in parallel to acquire multiple images; perform human body detection on each image to determine a number of human portraits; for each human portrait, cut off the upper part of the human body and extract the face image; extract the face features of the face image whose image quality score exceeds a first preset threshold; store each face feature in the Faiss database, and store the face image corresponding to the face feature and the timestamp of acquiring the face in the MySQL database; Step S2: when the periodic update time arrives or the trigger condition is met, the facial features of all unassigned facial identifiers are obtained from the Faiss database; and the facial features of all unassigned facial identifiers are sorted based on the timestamps corresponding to the facial features; For each sorted facial feature, the following operations are performed: Step S21: determining the image quality score corresponding to the facial feature; if the image quality score exceeds a second threshold, the facial feature is used as a facial feature to be processed, and the process proceeds to step S22; otherwise, the facial feature is processed; Step S22: comparing the facial features to be processed with the facial features stored in the Faiss database, and determining whether the facial features to be processed are new facial features based on the comparison results; If it is a newly appeared facial feature, assigning a facial identifier to the facial feature to be processed; Filling the face identifier corresponding to the face feature to be processed in the MySQL database; storing the face identifier of the face feature to be processed in the Redis database, recording the occurrence frequency of the face identifier as 1, and setting the expiration time corresponding to the face identifier; Otherwise, the face identifier corresponding to the face feature to be processed is completed in the MySQL database; a record of the face identifier is added in the Redis database, the face identifier of the face feature to be processed is recorded in the newly added record, and the occurrence frequency corresponding to the face identifier is updated; the face identifier corresponding to the face feature to be processed is completed in the Faiss database; When the facial identifier stored in the Redis database reaches its corresponding expiration time and no corresponding record is added, all records of the facial identifier are deleted, and the occurrence frequency corresponding to the facial identifier is cleared to zero, and the facial identifier is released; when the Redis database deletes the facial identifier, the MySQL database and the Faiss database delete all the contents corresponding to the facial identifier and clear them to zero, and the facial identifier is released; Step S3: obtaining facial features to be identified, and querying in the Faiss database whether there are facial features whose similarity with the facial features to be identified is greater than a third preset threshold; If so, obtaining the appearance frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than the third preset threshold, and when the appearance frequency exceeds the fourth preset threshold, determining that the person corresponding to the face feature to be identified is a high-frequency person; If not, the facial features to be identified are stored in the Faiss database to meet the triggering condition.

2. The method according to claim 1, characterized in that The step S1 comprises: Step S11: reading the stream addresses of multiple image acquisition devices through OpenCV, and obtaining unprocessed images from each image acquisition device; Step S12: Perform human body detection on each image using YOLOv5 to determine a number of portraits; for each portrait, cut off the upper half of the human body and extract the face image, and the face image is extracted in the following manner: determine that the length of the upper half of the human body is h1 and the width is w; cut off a part of size w*w from the upper left corner of the upper half of the human body as the face image or cut off a part of size w*w from the upper half of the human body from top to bottom. The part is taken as the face image; Step S13: extracting facial features of facial images whose image quality scores exceed a first preset threshold using a resnet50 network model, and normalizing all facial features; the image quality score is determined based on the SDD-FIQA score, illumination, facial posture, and facial clarity; Step S14: storing the normalized facial features and the timestamp of obtaining the portrait corresponding to the facial features in the MySQL database, wherein the timestamp is the time when the image acquisition device obtains the image of the person.

3. The method according to claim 1, characterized in that In the step S2, the face identifiers are allocated in ascending order according to currently available identifiers.

4. The method according to claim 1, characterized in that The step S3, obtaining the occurrence frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than a third preset threshold, includes: The Redis database, the MySQL database, and the Faiss database add and / or modify the relevant information corresponding to the face identifier corresponding to the face feature whose similarity to the face feature to be identified is greater than a third preset threshold; Determine the first timestamp when the facial feature to be identified is obtained and the facial identifier corresponding to the facial feature to be identified. If the interval time between other timestamps corresponding to the facial identifier stored in the MySQL database and the first timestamp is less than a second preset threshold, the frequency corresponding to the facial identifier remains unchanged; otherwise, the frequency corresponding to the facial identifier is increased by 1.

5. The method according to claim 1, characterized in that A whitelist is set in the Redis database. If the person who frequently enters and exits is a person on the whitelist, no alarm information is generated; otherwise, an alarm information is generated.

6. A high-frequency personnel counting device based on face recognition, characterized in that: The device comprises: Acquisition module: configured as multiple image acquisition devices working in parallel to acquire multiple images; perform human body detection on each image to determine a number of portraits; for each portrait, cut off the upper part of the human body and extract the face image; extract the face features of the face image whose image quality score exceeds a first preset threshold; store each face feature in the Faiss database, and store the face image corresponding to the face feature and the timestamp of acquiring the face image in the MySQL database; Sorting module: configured to obtain all facial features that are not assigned with facial identifiers from the Faiss database when the periodic update time arrives or the trigger condition is met; sort all facial features that are not assigned with facial identifiers based on the timestamps corresponding to the facial features; For each sorted facial feature, the following submodules are triggered: The first submodule is configured to determine the image quality score corresponding to the facial feature, and if the image quality score exceeds the second threshold, the facial feature is used as a facial feature to be processed, triggering the second submodule; otherwise, the facial feature is processed; The second submodule is configured to compare the facial features to be processed with the facial features stored in the Faiss database, and determine whether the facial features to be processed are newly appeared facial features based on the comparison results; If it is a newly appeared facial feature, a facial identifier is assigned to the facial feature to be processed; the facial identifier corresponding to the facial feature to be processed is completed in the MySQL database; the facial identifier of the facial feature to be processed is stored in the Redis database, the appearance frequency of the facial identifier is recorded as 1, and the expiration time corresponding to the facial identifier is set; Otherwise, the face identifier corresponding to the face feature to be processed is completed in the MySQL database; a record of the face identifier is added in the Redis database, the face identifier of the face feature to be processed is recorded in the newly added record, and the occurrence frequency corresponding to the face identifier is updated; the face identifier corresponding to the face feature to be processed is completed in the Faiss database; When the facial identifier stored in the Redis database reaches its corresponding expiration time and no corresponding record is added, all records of the facial identifier are deleted, and the occurrence frequency corresponding to the facial identifier is cleared to zero, and the facial identifier is released; when the Redis database deletes the facial identifier, the MySQL database and the Faiss database delete all the contents corresponding to the facial identifier and clear them to zero, and the facial identifier is released; Recognition module: configured to obtain facial features to be recognized, and query in the Faiss database whether there is a facial feature whose similarity with the facial features to be recognized is greater than a third preset threshold; If so, obtaining the appearance frequency of the face identifier corresponding to the face feature whose similarity with the face feature to be identified is greater than the third preset threshold, and when the appearance frequency exceeds the fourth preset threshold, determining that the person corresponding to the face feature to be identified is a high-frequency person; If not, the facial features to be identified are stored in the Faiss database to meet the triggering condition.

7. A computer-readable storage medium, wherein a plurality of instructions are stored in the storage medium; the plurality of instructions are used for a processor to load and execute the method as claimed in any one of claims 1 to 5.

8. An electronic device, characterized in that: The electronic device comprises: A processor, which is used to execute multiple instructions; A memory for storing a plurality of instructions; The plurality of instructions are used to be stored in the memory and loaded and executed by the processor according to any one of claims 1 to 5.

Images