Face recognition method and system for one-unlocking multi-entry of intelligent lock

Abstract

The invention discloses a face recognition method and system of an intelligent lock with one-unlocking and multi-entry, and the method comprises the steps: extracting the face information and human body information of a multi-entry person in a snapshot time period; if the human face image and the human body image are extracted from the same person image at the same time, direct pairing succeeds; if the unique pairing relationship is determined through an exclusion method, the pairing is successful; the similarity of the human body information is calculated by retrieving the one-open multi-input track list, and if the similarity is larger than a preset threshold value, pairing succeeds; constructing a spatial-temporal trajectory sequence of the unretrieved and successfully paired human body information, and carrying out compatibility calculation on the spatial-temporal trajectory sequence and a human face clustering spatial-temporal trajectory sequence in a one-open multi-input trajectory list to determine a confidence coefficient; and fusing the pairing results, and for the face information corresponding to each piece of human body image information, taking the face information with the highest face matching confidence as a fusion result. By using the embodiment of the invention, the identification of the result of the multi-entry personnel with confidence can be constructed, and the identification of the multi-entry personnel is realized.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent locks, and particularly relates to a face recognition method and system for an intelligent lock with one open and multiple entries. Background Art

[0002] With the acceleration of the urbanization process, the number of migrant people in cities is increasing. Rental houses and apartments are also undergoing a transformation. Public security agencies in many places are promoting the installation of intelligent locks in rental houses / apartments. Intelligent locks are gradually being introduced into the management of rental houses / apartments and residential houses, bringing many conveniences to the management agencies of rental houses / apartments and local public security management departments, and improving the safety and orderliness of rental houses / apartments.

[0003] As a security prevention facility, the intelligent lock provides security protection for residents, and at the same time provides the management agency with records of identifying the entry and exit of residents. Currently, the technologies for intelligent locks to identify the entry and exit of people are mainly fingerprint comparison and face recognition comparison. In order to make full use of video images to monitor the situation at the entrance, the intelligent lock installed at the entrance of the household will be equipped with one or more cameras. The personnel video images collected on-site are shared with other intelligent locks or can be aggregated to the management platform of the intelligent lock. However, currently, the intelligent lock can only accurately identify the unlocking personnel. The personnel entering and exiting together with the unlocking personnel cannot be effectively identified due to non-cooperative unlocking recognition, the shooting angle of the door lock camera, personnel standing positions, occlusion, etc., becoming a monitoring blind spot for security control. Summary of the Invention

[0004] The purpose of the present invention is to provide a face recognition method and system for an intelligent lock with one open and multiple entries to solve the deficiencies in the prior art, and be able to construct an identification of the results of multiple entering personnel with confidence, and realize the identification of multiple entering personnel.

[0005] An embodiment of the present application provides a face recognition method for an intelligent lock with one open and multiple entries, and the method includes: Detection of multiple entering personnel: Extract the face information and body information of multiple entering personnel during the capture time period; Multi-mode pairing: Direct pairing and exclusion pairing: If a face image and a body image are simultaneously extracted from the same personnel image, the direct pairing is successful, and the confidence level is the determined level; If the unique pairing relationship is determined by the exclusion method, the pairing is successful, and the confidence level is the determined level; Face retrieval pairing: For the body information that has not been successfully paired directly or by exclusion, calculate the similarity of the body information by retrieving the one-open-multiple-entry trajectory list. If the similarity is greater than the preset threshold, the pairing is successful, and the confidence level is medium or high; Space-time trajectory compatibility pairing: For the body information that has not been successfully paired by retrieval, construct its space-time trajectory sequence, and perform compatibility calculation with the face clustering space-time trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence level; Pairing result fusion: fuse each pairing result, and for the face information corresponding to each human body image information, select the one with the highest face matching confidence as the fusion result.

[0006] Optionally, the direct pairing includes: Based on the video image captured by the smart lock, use the face detection algorithm to extract the face image / feature vector, and at the same time use the human body detection algorithm to extract the human body image / structured feature; If the face image / feature vector and the human body image / structured feature are simultaneously extracted from the same person's image, it is directly determined that the pairing is successful, and the confidence level is the certainty level; If the pairing is not completely successful by the direct pairing method, then use the exclusion pairing method to uniquely pair the remaining face images / feature vectors and human body images / structured features. If the remaining face images / feature vectors and human body images / structured features are in a unique pairing relationship, the pairing is successful, and the confidence level is the certainty level.

[0007] Optionally, the face retrieval pairing includes: For the human body image / structured feature that is not directly paired successfully, calculate its similarity with the human body image / structured feature in the one-open-multiple-entry trajectory list stored in the smart lock by retrieving the list; If the similarity is greater than the preset threshold (80%), then extract the corresponding face image / feature vector from the one-open-multiple-entry trajectory list, and the pairing is successful, and the confidence level is medium or high; Update the paired face image / feature vector and the confidence level to the current one-open-multiple-entry event information.

[0008] Optionally, the spatio-temporal trajectory compatibility pairing includes: Based on the human body image / structured feature that is not paired successfully, construct its spatio-temporal trajectory sequence, including the time trajectory sequence and the space trajectory sequence; Compare the constructed spatio-temporal trajectory sequence with the face clustering spatio-temporal trajectory sequence in the one-open-multiple-entry trajectory list to determine whether they are completely coincident, partially coincident or completely compatible; If the spatio-temporal trajectories are completely coincident, the confidence level is the certainty level; if they are partially coincident and completely compatible, the confidence level is high; if they are not coincident but completely compatible, the confidence level is medium; Update the paired face image / feature vector and the confidence level to the current one-open-multiple-entry event information.

[0009] Optionally, the pairing result fusion includes: Fuse each pairing result, and select the face image / feature vector with the highest confidence level corresponding to each human body image / structured feature; Generate complete one-open-multiple-entrance event information, where the one-open-multiple-entrance event information includes an event serial number, a door lock ID, a time, an unlocker number, multiple co-traveler face images / feature vectors, multiple co-traveler face matching confidence levels, and multiple co-traveler structured features; Send the one-open-multiple-entrance event information to other intelligent locks in a multicast form and to the platform in a unicast form at the same time; Write the one-open-multiple-entrance event information into the one-open-multiple-entrance trajectory list of the intelligent lock, with a storage time of 10 minutes and automatic deletion after expiration.

[0010] Another embodiment of the present application provides a face recognition system for one-open-multiple-entrance of an intelligent lock, and the system includes: A monitoring module for detecting multiple entering persons: extracting the face information and body information of multiple entering persons during the capture time period; A pairing module for multi-mode pairing: direct pairing and exclusion pairing: if a face image and a body image are simultaneously extracted from the same person image, the direct pairing is successful and the confidence level is the determined level; if the unique pairing relationship is determined by the exclusion method, the pairing is successful and the confidence level is the determined level; face retrieval pairing: for the body information that has not been directly paired or successfully paired by exclusion, calculate the similarity of the body information by retrieving the one-open-multiple-entrance trajectory list. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; spatio-temporal trajectory compatibility pairing: for the body information that has not been successfully paired by retrieval, construct its spatio-temporal trajectory sequence and perform compatibility calculation with the spatio-temporal trajectory sequence of face clustering in the one-open-multiple-entrance trajectory list to determine the confidence level; A fusion module for fusing pairing results: fusing each pairing result, and taking the face matching confidence level with the highest value as the fusion result for the face information corresponding to each body image information.

[0011] Another embodiment of the present application provides a storage medium, in which a computer program is stored, and the computer program is set to execute the method described in any one of the above when running.

[0012] Another embodiment of the present application provides an electronic device, including a memory and a processor, where a computer program is stored in the memory, and the processor is set to run the computer program to execute the method described in any one of the above.

[0013] Compared with the prior art, a face recognition method for one-open-and-multiple-entry of an intelligent lock provided by the present invention extracts face information and human body information of multiple entering persons during the capture time period. If a face image and a human body image are simultaneously extracted from the same person's image, the pairing is directly successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the one-open-and-multiple-entry trajectory list and calculating the similarity of the human body information, if the similarity is greater than the preset threshold, the pairing is successful; for the human body information that fails to be retrieved and paired successfully, its spatio-temporal trajectory sequence is constructed and compatibility calculation is performed with the spatio-temporal trajectory sequence of face clustering in the one-open-and-multiple-entry trajectory list to determine the confidence level; the various pairing results are fused, and for the face information corresponding to each human body image information, the one with the highest face matching confidence level is taken as the fusion result, so as to be able to construct an identification of the result of multiple entering persons with confidence and realize the identification of multiple entering persons. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] Figure 1 FIG. is a hardware structure block diagram of a computer terminal for a face recognition method for one-open-and-multiple-entry of an intelligent lock provided by an embodiment of the present invention; Figure 2 FIG. is a schematic flowchart of a face recognition method for one-open-and-multiple-entry of an intelligent lock provided by an embodiment of the present invention; Figure 3 FIG. is a schematic structural diagram of a face recognition system for one-open-and-multiple-entry of an intelligent lock provided by an embodiment of the present invention. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0015] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention and should not be construed as limiting the present invention.

[0016] An embodiment of the present invention first provides a face recognition method for one-open-and-multiple-entry of an intelligent lock, which can be applied to an electronic device, such as a computer terminal, specifically, a general computer, etc.

[0017] The following takes running on a computer terminal as an example to describe it in detail. Figure 1 FIG. is a hardware structure block diagram of a computer terminal for a face recognition method for one-open-and-multiple-entry of an intelligent lock provided by an embodiment of the present invention. As Figure 1 shown, the computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory may include a non-volatile storage medium and an internal memory.

[0018] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions, and when the program instructions are executed, the processor can execute any face recognition method for one-open-and-multiple-entry of an intelligent lock.

[0019] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0020] The internal memory provides an environment for the operation of computer programs in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any one of the face recognition methods for one intelligent lock with multiple entries.

[0021] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 1 the structure shown in is only a block diagram of some structures related to the solution of this application, and does not constitute a limitation on the computer device to which the solution of this application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0022] It should be understood that the processor can be a central processing unit (CPU), and the processor can also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc.

[0023] See Figure 2 , the embodiments of the present invention provide a face recognition method for one intelligent lock with multiple entries, which may include the following steps: S201, multi-entry personnel detection: within the capture time period, extract the face information and body information of multi-entry personnel; S202, multi-mode pairing, including: Direct pairing and exclusion pairing: If a face image and a body image are simultaneously extracted on the same personnel image, the direct pairing is successful, and the confidence level is the certainty level; if the unique pairing relationship is determined by the exclusion method, the pairing is successful, and the confidence level is the certainty level; specifically, the direct pairing includes: According to the video image captured by the intelligent lock, use the face detection algorithm to extract the face image / feature vector, and at the same time use the body detection algorithm to extract the body image / structured feature; If a face image / feature vector and a body image / structured feature are simultaneously extracted on the same personnel image, it is directly determined that the pairing is successful, and the confidence level is the certainty level; If the direct pairing method fails to achieve complete pairing, the exclusion pairing method is used to uniquely pair the remaining face images / feature vectors and body images / structured features. If the remaining face images / feature vectors and body images / structured features have a unique pairing relationship, the pairing is successful and the confidence level is of the definite level.

[0024] Face retrieval pairing: For the body information that has not been directly paired or successfully paired by exclusion, by retrieving a one-open-multiple-entry trajectory list, the similarity of the body information is calculated. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high. Specifically, the face retrieval pairing includes: For the body image / structured feature that has not been directly paired successfully, by retrieving the one-open-multiple-entry trajectory list stored in the intelligent lock, its similarity with the body image / structured feature in the list is calculated; If the similarity is greater than the preset threshold (80%), the corresponding face image / feature vector is extracted from the one-open-multiple-entry trajectory list, the pairing is successful, and the confidence level is medium or high; The successfully paired face image / feature vector and confidence level are updated to the current one-open-multiple-entry event information.

[0025] Space-time trajectory compatibility pairing: For the body information that has not been successfully paired by retrieval, its space-time trajectory sequence is constructed and compatibility calculation is performed with the face clustering space-time trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence level. Specifically, the space-time trajectory compatibility pairing includes: According to the body image / structured feature that has not been successfully paired, its space-time trajectory sequence is constructed, including a time trajectory sequence and a space trajectory sequence; The constructed space-time trajectory sequence is compared with the face clustering space-time trajectory sequence in the one-open-multiple-entry trajectory list to determine whether they completely coincide, partially coincide, or are completely compatible; If the space-time trajectories completely coincide, the confidence level is of the definite level; if they partially coincide and are completely compatible, the confidence level is high; if they do not coincide but are completely compatible, the confidence level is medium; The successfully paired face image / feature vector and confidence level are updated to the current one-open-multiple-entry event information.

[0026] S203, pairing result fusion: The various pairing results are fused. For the face information corresponding to each body image information, the one with the highest face matching confidence level is taken as the fusion result. Specifically, the pairing result fusion includes: The various pairing results are fused, and the face image / feature vector with the highest confidence level corresponding to each body image / structured feature is taken; Generate complete one-open-multiple-entrance event information, where the one-open-multiple-entrance event information includes an event serial number, a door lock ID, a time, an unlocker number, multiple co-traveler face images / feature vectors, multiple co-traveler face matching confidence levels, and multiple co-traveler structured features; Send the one-open-multiple-entrance event information to other intelligent locks in the form of multicast, and at the same time send it to the platform in the form of unicast; Write the one-open-multiple-entrance event information into the one-open-multiple-entrance track list of the intelligent lock, with a storage time of 10 minutes, and it will be automatically deleted after expiration.

[0027] Intelligent door lock: Also known as an intelligent lock, it refers to a lock improved on the basis of a traditional mechanical lock, which is more intelligent and simpler in terms of user security, identification, and management. The intelligent door lock mentioned in this article has the function of opening the door by face recognition.

[0028] Currently, the method of face target clustering is generally often used. For example, T / CSPIA 008-2022 "Technical Requirements for Video Image Target Clustering Services" mentions clustering and archiving video target objects collected from a large range of video images to build a video image target archive. There are mainly two forms of clustering: full-scale clustering at regular intervals and incremental clustering each time a target is added. This clustering method requires building a very large GPU computing cluster to perform clustering calculations on massive data. If used for the identification of multiple entrants in a one-open-multiple-entrance situation in an intelligent lock, it is impossible to cluster because the information of the communicator is not provided in the information reported by each intelligent lock.

[0029] The purpose of the present invention is to provide a method for identifying the faces of multiple entrants whose faces are not recognized in a one-open-multiple-entrance situation, sharing the one-open-multiple-entrance recognition track points among intelligent locks with the multicast method, and realizing the face recognition of all personnel at the time of one-open-multiple-entrance in the message reported by the intelligent lock to the platform. By storing the one-open-multiple-entrance information detected and recognized by each intelligent lock through multicast sharing in each intelligent lock, the method of combining spatio-temporal trajectory compatibility and human body image structured features is used to pair the faces of multiple entrants whose faces are not recognized in a one-open-multiple-entrance situation, and the confidence level of the face pairing is marked with a three-level confidence level, providing a reference for the pairing of other door locks and also providing support for platform applications. In practical applications, the complete process of a technology is as follows: (1) Prerequisites: Install intelligent locks on the household entrance doors (each door lock has an independent ID, door lock IP address, door lock geographical location, configured multicast group IP address, etc.), and connect to the intelligent lock management platform (hereinafter referred to as the platform), register with the platform, and can receive remote configuration of the multicast group IP address by the platform network management tool, and periodically update the list of registered residents from the platform (data items include: personnel number, name, ID card number, face image / feature vector); the intelligent locks can be connected to other intelligent locks through multicast.

[0030] The platform records the list of smart locks, including door lock ID, door lock IP address, door lock geographic location, multicast group and multicast group IP address; the platform configures the smart locks of a building or a unit as a multicast group according to the building or unit situation in the community. Smart door lock: also called smart lock, refers to a lock that is different from traditional mechanical locks and is more intelligent and simple in terms of user security, identification, and management. The smart door lock referred to in this article has the function of opening the door by face recognition. A few concept explanations: 1) Multiple entries at once refers to the phenomenon that when a resident registers to check in and unlocks the door, multiple people enter with him. Due to the installation position of the door lock camera and the occlusion between multiple people, it is impossible to capture the facial images of all the people, resulting in the inability to identify all the people who enter at once; 2) Multiple-entry personnel refers to the personnel who enter the room together with the registered resident after unlocking the door. If the registered resident unlocks the door and enters alone, it can be regarded as a special case of multiple-entry personnel; 3) Capture time period: refers to the time period from when multiple people enter the door through the camera shooting area to when all of them enter the door for capture and identification.

[0031] 4) One-open-multiple-entry trajectory list: The information of one-open-multiple-entry events collected by the smart lock or sent by other door locks through multicast, including event sequence number, door lock ID, time, unlocker number, multiple peer face images / feature vectors, multiple peer face matching confidences, and multiple peer structured features.

[0032] (2) Face matching confidence rules: three confidence levels are set according to different situations The confidence level in this article is used to indicate the possibility that a certain face image / feature vector extracted from a video image and a certain body image / structured feature are the same person. The higher the confidence level, the higher the possibility that the face image and the body image are the same person. It is mainly used to describe the credibility of the face image found by the person who only captured the body image. The confidence level is divided into three levels: certain confidence level, high confidence level, and medium confidence level.

[0033] Deterministic confidence: When the smart lock captures both the face and body images at the same time, the pairing is successful, or at least one of the received trajectory points has a certain confidence level for the body image (structured features), or the time-space trajectory sequence is completely overlapped.

[0034] High confidence: If the spatiotemporal trajectory sequences partially overlap and are completely compatible; or if the spatiotemporal trajectory sequences do not overlap but are completely compatible, and the person corresponding to the face trajectory is the unlocked person.

[0035] Medium confidence: Or the spatio-temporal trajectory sequences do not coincide, but are completely compatible.

[0036] Face image / feature vector: It refers to the image of the face area extracted from the video image captured by the intelligent lock camera according to the face intelligent detection, recognition and segmentation algorithm, and the feature vector of the face image is extracted.

[0037] Human body image / structured features: It refers to the image of the human body area extracted from the video image captured by the intelligent lock camera according to the personnel intelligent detection, recognition and segmentation algorithm, and the structured features of the human body image are extracted by the structured analysis algorithm. Here, the human body refers to the entire appearance of the person, and the image quality of the face part does not meet the requirements of face recognition.

[0038] Spatio-temporal trajectory point: It refers to the current clock (i.e., time position) and the geographical location of the intelligent lock when capturing and extracting the face image / feature vector and the human body image / structured features (note: that is, the spatial position, represented by the intelligent lock ID number, and the intelligent lock ID is associated with a clear geographical location). The spatio-temporal trajectory points can be sorted by time to form a time trajectory sequence and sorted by geographical location to form a spatial trajectory sequence. The spatio-temporal trajectory points in this article refer to the trajectory points collected during the period from the start to the end of a one-open-multiple-entry event.

[0039] Compatibility of trajectory point sequences: It means that two out-of-control trajectory point sequences have no overlapping trajectory points in the time trajectory sequence and the spatial trajectory sequence, but exist in the intervals of the time trajectory sequence and the spatial trajectory sequence. For example: Suppose the door locks A - Z are sorted in spatial order (route), and the time points A - Z are sorted in time order. A certain piece of structured information is captured by (door lock A, time point A), (door lock C, time point C), (door lock E, time point E). A certain face is captured by (door lock B, time point B), (door lock D, time point D), (door lock F, time point F). Although the structured information and the face have no common spatio-temporal overlapping points, they are obviously compatible and there is a high probability of consistency.

[0040] (3) Perform face pairing for multiple entering persons during the door lock capturing period: The intelligent lock intelligently tracks and recognizes the faces and human body images of multiple entering persons through video images, and performs face and human body pairing; Detection and recognition of faces and human bodies of multiple entering persons: During the capturing period of the intelligent door lock, face images (feature vectors are extracted through intelligent analysis) and human body images (structured features are extracted after structured analysis) are extracted through intelligent detection and recognition technologies.

[0041] Example: For instance, when intelligent lock A detects and identifies multiple people entering through a single access point, the registered occupant a of household A undergoes face recognition comparison and passes the comparison by A and unlocks the lock. At this time, there are other accompanying people (such as b, c, d) entering the door with a. However, due to factors such as the installation position and angle of the camera of the intelligent lock and occlusion among people, only the faces of b and c are captured, while the face of d is not captured. Nevertheless, the body images of all people (a, b, c, d) can be collected.

[0042] There are 2 methods for pairing the face images / feature vectors and body images / structured features of multiple people entering during the captured time period: the direct pairing method and the exclusion pairing method.

[0043] Direct pairing method: If both face images / feature vectors and body images / structured features are extracted from the same person image, it is determined that the pairing is successful, and the confidence level is the definite level (judged according to the confidence level determination rule in step (2)).

[0044] Example: If the intelligent lock captures a full-body image of a person, which includes both the face and the complete appearance of the person, at this time, face images / feature vectors and body images / structured features can be respectively extracted, and they are extracted from the image of the same person, so it is obviously the same person.

[0045] Exclusion pairing method: Among the multiple face images / feature vectors and body images / structured features extracted when multiple people enter through a single access point, if there are 2 people and one person (a pair of face images / feature vectors and body images / structured features) is successfully paired, then the remaining pair of face images / feature vectors and body images / structured features can be directly paired successfully as another person; similarly, in the case of multiple people, after direct pairing is successful, if there is only one remaining pair of face images / feature vectors and body images / structured features, it can be directly paired successfully as another person. If there are multiple remaining face images / feature vectors and body images / structured features, they cannot be directly paired. The confidence level of successful pairing using the exclusion method is the definite level.

[0046] Example: It is illustrated by taking the cases of 2 people and multiple people as examples.

[0047] Case of 2 people: If the face images / feature vectors (F1, F2) and body images / structured features (V1, V2) collected during the time period when multiple people enter through a single access point are such that it is already clear that F1 and V2 are successfully paired by the direct pairing method, then the remaining F2 and V1 can be paired successfully. Since there are only 2 people, the remaining one can only be the second person; In the case of multiple people: Taking three people as an example, if the face images / feature vectors collected within the one-open-multiple-entry event segment are (F1, F2, F3), and the body images / structured features are (V1, V2, V3), and if it has been determined that the direct pairing method between F1 and V2, and F2 and V1 is successful, then the remaining F3 and V3 can be successfully paired. Since there are only three people, the remaining one can only be the third person; by analogy, as long as there is only one person left, and only one pair of face images / feature vectors and body images / structured features, then the remaining pair of face images / feature vectors and body images / structured features can be successfully paired.

[0048] If the confidence level that all multiple-entry personnel are successfully paired to faces within the one-open-multiple-entry time period is at the determined level, then skip steps (4) and (5), directly write into the one-open-multiple-entry trajectory list of the door lock, and at the same time send it to other door locks for one-open-multiple-entry trajectory list synchronization via multicast and send it to the platform in unicast form for platform applications.

[0049] (4) Conduct face retrieval and pairing for multiple-entry personnel in the door lock storage record: Use the body images / structured features that were not successfully paired to face images in step (3) to compare and retrieve the body images / structured features in the one-open-multiple-entry trajectory list to pair with face images / feature vectors; Conduct comparison and retrieval in the one-open-multiple-entry trajectory list. According to the similarity calculation of the body images / structured features, use the face images / feature vectors and face pairing confidence levels in the one-open-multiple-entry trajectory list corresponding to the body image / structured feature with a similarity value greater than the threshold (usually set to 80%) and the largest. If the similarity values are all less than the threshold, the retrieval and pairing fail.

[0050] Note: The one-open-multiple-entry trajectory list synchronization of the intelligent door lock stores the one-open-multiple-entry event information of each member of the multicast group; before writing the one-open-multiple-entry event information captured and collected into the one-open-multiple-entry trajectory list of this door lock, sending a multicast message, and sending a message to the platform, it is necessary to identify the faces of all multiple-entry personnel (there may be cases with low confidence levels); if the similarity values are all less than the threshold, it means that the similarity of the body images / structured features is low, and the possibility of the same person is low. If the face image is paired at this time, there will be a large error.

[0051] (5) Conduct compatibility pairing of the spatio-temporal trajectories of multiple-entry personnel in the door lock storage record: Calculate the compatibility of the spatio-temporal trajectory sequence of multiple-entry personnel constructed with body images / structured features and the spatio-temporal trajectory sequence after face clustering, and conduct pairing and determine the face pairing confidence level according to the face pairing confidence level rule.

[0052] Face clustering and spatio-temporal trajectory sequence construction: Perform clustering calculations on all face images / feature vectors based on a one-open-and-multiple-entry trajectory list, and construct a spatio-temporal trajectory sequence for the clustered trajectory points (Note: Trajectory points are records of one-open-and-multiple-entry events with spatio-temporal characteristics) in the order of time (timestamp) and space (door lock ID number).

[0053] Example: The clustering algorithm for video images calculates the similarity between face images / feature vectors and archives those with similarity meeting the threshold into one class, corresponding to one person. Suppose there are face images / feature vectors of faces a, b, c, d, ……, z in the one-open-and-multiple-entry trajectory list stored in the door lock. For face a, there are n trajectory points a1, a2, a3, ……, an after clustering (similarly for faces b, ……, z). Construct a time trajectory sequence T1, T2, ……, Tn according to the timestamp and a space trajectory sequence W1, W2, ……, Wn according to the order of the door lock ID numbers.

[0054] Retrieval of human body images / structured features of multiple-entry personnel and spatio-temporal trajectory sequence construction: Retrieve and compare the human body images / structured features of multiple-entry personnel with all human body images / structured features in the one-open-and-multiple-entry trajectory list, extract all trajectory points with similarity ≥ threshold (usually set to 80%), and construct a spatio-temporal trajectory sequence according to the order of time (timestamp) and space (door lock ID number).

[0055] Example: Suppose the human body image / structured feature of multiple-entry personnel is x. After calculating the similarity with all human body images / structured features in the one-open-and-multiple-entry trajectory list, m trajectory points S1, S2, ……, Sm with similarity ≥ 80% are extracted. Construct a time trajectory sequence t1, t2, ……, tm for these m trajectory points according to the timestamp and a space trajectory sequence w1, w2, ……, wm according to the order of the door lock ID numbers.

[0056] Compatibility calculation of spatio-temporal trajectory sequences: Compare each trajectory point of the time and space trajectory sequences of multiple-entry personnel with the time and space trajectory sequences of face clustering one by one to check whether the time, space, and order relationships are consistent. If the spatio-temporal trajectory point sequences are the same, it means the trajectory points coincide; if the spatio-temporal trajectory points do not coincide, but the trajectory points are in the middle of the other's spatio-temporal trajectory sequence, it means the trajectories are compatible.

[0057] Explanation of spatio-temporal trajectory sequence: Suppose the time trajectory sequence of the human body image / structured feature x of multiple-entry personnel is t1, t2, ……, tm, and the spatial trajectory sequence is w1, w2, ……, wm. Compare t1, t2, ……, tm with the time trajectory sequence T1, T2, ……, Tn of the face a in the face clustering one by one. If t1 = T1, t2 = T2, ……, tm = Tm, it means that they coincide at each time point, indicating that the time trajectory sequences coincide. Similarly, the comparison can also be carried out for the spatial sequence.

[0058] Example of the compatibility of trajectory sequences: Suppose the time trajectory sequence of the human body image / structured feature x of multiple-entry personnel is t1, t3, t5, t7, and the time trajectory sequence of the face a is T2, T4, T6. If t1 ≤ T2 ≤ t3, t3 ≤ T4 ≤ t5, t5 ≤ T6 ≤ t7, then the time trajectory sequence of x is compatible with the time trajectory sequence of a. Compatibility includes partial compatibility and complete compatibility. Complete compatibility means that the time trajectory sequence of x can be completely inserted into the time gaps of the time trajectory sequence of a, and partial compatibility means that part of the time trajectory sequence can be inserted into the time gaps of the other party.

[0059] Determining the face matching confidence level based on the compatibility of spatio-temporal trajectory sequences: It is divided into several situations, as shown in Table 1 below.

[0060] Table 1

[0061] Explanation of determining the face matching confidence level based on the compatibility of spatio-temporal trajectory sequences: 1) Complete coincidence of spatio-temporal trajectories: That is, the spatio-temporal trajectory sequence of multiple-entry personnel coincides completely with the spatio-temporal trajectory sequence after face clustering of a certain person, then the confidence level is the determined level; Explanation: This situation generally occurs when a person's face and body image are captured in front of multiple door locks.

[0062] 2) Partial coincidence of spatio-temporal trajectories and complete compatibility: That is, several of the spatio-temporal trajectory sequences of multiple-entry personnel coincide with the spatio-temporal trajectory sequence after face clustering of a certain person, and all other trajectory points are compatible. At this time, the confidence level is a high confidence level; Explanation: This situation generally occurs when a person's face and body image are captured simultaneously in front of multiple door locks, and there are also some cases where only the face or body image is captured.

[0063] 3) Non-coincidence of spatio-temporal trajectories, but complete compatibility and the face trajectory sequence is the unlocking person: If it is completely compatible, and the face in the spatio-temporal trajectory of the face is a certain unlocking person (that is, the registered person for a certain household resident), at this time, the confidence level is a high confidence level; Explanation: This situation generally occurs when only the face or body image of a person is captured in front of multiple door locks, but it can be determined that it is a person who directly unlocks and opens the door. It may be a neighbor visiting.

[0064] 4) The spatio-temporal trajectories do not overlap but are completely compatible: If the spatio-temporal trajectories are completely compatible, the confidence level is medium confidence at this time; Explanation: This situation generally occurs when only the face or body image of a person is captured in front of multiple door locks, and the possibility of it being the same person is relatively high.

[0065] 5) The spatio-temporal trajectories do not overlap, and are partially compatible or incompatible: Only a few trajectory points are compatible, or even there are no compatible trajectory points. At this time, the confidence level is no confidence, that is, not confident; Explanation: This situation generally occurs when the person does not appear in front of other door locks, or because of similar clothing and appearance, but is not the same person. At this time, it is meaningless to use compatibility for confidence.

[0066] The cross-lock supplementary method is used to further correct the pairing result: For smart lock A (assuming that there are multiple incoming people who open one and enter multiple times in A and their faces are not detected and recognized, and through the query and retrieval of the open-one-and-multiple-incoming trajectory list or compatibility calculation for pairing, and the confidence level of the pairing does not reach high confidence level. In order to further improve the face paired to high confidence level, a surveillance method can be adopted to wait for the person to have a higher face matching confidence level at other door locks). Wait for no more than 10 minutes. When receiving the face and its confidence level of the un-detected multiple incoming people from B, compare the confidence level calculated by A before with the confidence level given by B just received, and take the face with the higher confidence level of the two as the final face and its confidence level of the un-detected multiple incoming people.

[0067] Example: For example, in the open-one-and-multiple-incoming event with the serial number XXX01 of A, there are 4 people (a, b, c, d), where b is a multiple incoming person once and the face is not captured. The result calculated by A according to the structured features of b through the trajectory compatibility method is face F1 (the face image / feature vector of F1 is extracted from the open-one-and-multiple-incoming trajectory list stored in A, and the calculated confidence level is recorded as medium). During the 10-minute retention period, A receives the multicast message YYY01 from B. B also recognizes the appearance of b, and according to the matching calculation of B, the confidence level of b corresponding to F2 is high. At this time, A finds that B provides a face image / feature vector F2 of b with a higher confidence level, then updates the F2 given by B to its own open-one-and-multiple-incoming trajectory list, and updates the face image / feature vector of b in the open-one-and-multiple-incoming event of XXX01 to F2.

[0068] (6) Fusion and processing of face pairing results for one-open-multiple-entry events: The intelligent door lock fuses the face pairing results completed in steps (3), (4), and (5), and for each face image / feature vector corresponding to a human body image / structured feature, the one with the highest face pairing confidence is taken as the fusion result.

[0069] The intelligent door lock performs fusion based on face pairing confidence: For each multi-entry person, the face pairing results completed in steps (3), (4), and (5) are compared, and the corresponding face image / feature vector and face pairing confidence with the highest face pairing confidence are taken as the fusion result.

[0070] Example: Suppose door lock A captures four people, a, b, c, and d. a is the unlocking person, and b, c, and d are accompanying people, that is, multi-entry people. If b's face image and human body image are not directly captured, during step (3), when performing face pairing for multi-entry people within the capture time period of the door lock, the face pairing confidence for the face image / feature vector x paired with b is at the determined level; during step (4), when performing face retrieval and pairing for multi-entry people in the door lock storage record, the face pairing confidence for the face image / feature vector y paired with b is at the intermediate level; during step (5), when performing spatio-temporal trajectory compatibility pairing for multi-entry people in the door lock storage record, the face pairing confidence for the face image / feature vector z paired with b is at the intermediate level. Then, the paired face image / feature vector for b should be determined as x, and its face pairing confidence is at the determined level.

[0071] After the intelligent door lock completes face pairing, it forms a complete one-open-multiple-entry event information and writes it into the one-open-multiple-entry trajectory list of the door lock. At the same time, it is sent to other door locks and the platform via multicast. The data items include event serial number, door lock ID, time, unlocking person number, face images / feature vectors of multiple accompanying people, face pairing confidences of multiple accompanying people, structured features of multiple accompanying people, etc. The data in the one-open-multiple-entry trajectory list is stored for 10 minutes.

[0072] An example of the data items for one-open-multiple-entry events is as follows: Door lock A captures four people, a, b, c, and d. a is the unlocking person, and b, c, and d are accompanying people, that is, multi-entry people.

[0073] Event serial number: The ID of A + unlocking time; Door lock ID: Extract the ID of A; Time: Extract the current clock of A; Unlocking person number: The person number of a, extracted from the list of registered residents of A; Unlocking person's face image / feature vector: The face image / feature vector of a; Unlocking person's structured feature: Extracted by performing structured intelligent analysis based on a; Face image / feature vector of companion 1: b face image / feature vector; Confidence level of face image / feature vector of companion 1: Graded according to three-level confidence level; Structured features of companion 1: Extracted by structured intelligent analysis based on b; Face image / feature vector of companion 2: c face image / feature vector; Confidence level of face image / feature vector of companion 2: Graded according to three-level confidence level; Structured features of companion 2: Extracted by structured intelligent analysis based on c; Face image / feature vector of companion 3: d face image / feature vector; Confidence level of face image / feature vector of companion 3: Graded according to three-level confidence level; Structured features of companion 3: Extracted by structured intelligent analysis based on d; Multicast message sending: According to the multicast group IP address configured uniformly by the platform, send the information of one-open-multiple-entry events in the form of multicast; The payload of the multicast packet includes event serial number, door lock ID, time, unlocker number, multiple face images / feature vectors of companions, multiple face matching confidence levels of companions, multiple structured features of companions, etc.; Example: Assume that intelligent lock A detects and identifies persons a, b, c, d (where a is the registered occupant, and b, c, d are companions), and sends out the face images / feature vectors, face matching confidence levels, structured features and other information of the four persons; The confidence level of face matching is sent to other intelligent locks at the same time for reference by other intelligent locks.

[0074] (7) Multicast message sending and receiving: After the intelligent door lock completes face pairing, it sends the one-open-multiple-entry event information that has been paired by this lock to other door locks in the form of multicast. Other door locks receive and extract the one-open-multiple-entry event information in the multicast message, write it into the one-open-multiple-entry trajectory list of this door lock, and compare the human body image / structured features with the one-open-multiple-entry trajectory list of this door lock. If the face matching confidence level is higher than the trajectory point in the one-open-multiple-entry trajectory list, update the face image / feature vector and face matching confidence level of the trajectory point with the maximum human body image / structured feature similarity value in the one-open-multiple-entry trajectory list.

[0075] Multicast Message Reception and Processing: After receiving a multicast message, the intelligent door lock within the multicast group extracts and stores the one-open-multiple-entry event information. It calculates the similarity between the human body images / structured features with high or certain-level face matching confidence in the multicast message and the human body images / structured features in the one-open-multiple-entry trajectory list of this lock. If the face matching confidence in the multicast message is higher, the face image / feature vector corresponding to the one-open-multiple-entry trajectory list with the largest similarity value greater than the threshold (usually set to 80%) and the face matching confidence are updated to the face image / feature vector and face matching confidence in the multicast message. The one-open-multiple-entry event information extracted from the multicast message is written into the one-open-multiple-entry trajectory list of this door lock (data items include event serial number, door lock ID, time, unlocker number, multiple peer human face images / feature vectors, multiple peer human face matching confidences, multiple peer human structured features, etc.), and the storage time is usually set to 10 minutes and will be automatically deleted after the storage expiration date.

[0076] Explanation: After calculating the similarity of the human body images / structured features, it is required that the similarity value is greater than the threshold (usually set to 80%) and is the largest, in order to update the person with the highest similarity of the human body image in the one-open-multiple-entry trajectory list; since the similarity threshold does not meet the requirements, it may not be the same person, so it needs to be excluded; not updating all the trajectory points that meet the threshold requirements is because as long as one trajectory point is updated, the face image / feature vector with the highest confidence can be sufficiently paired during the face retrieval and pairing of multiple incoming personnel for this door lock; the multicast message does not need to reply about the update situation of this door lock because each door lock has synchronized all the spatio-temporal trajectory points of the multicast group members and can update them separately; moreover, in principle, there should be no situation where the face matching confidence in the newly received multicast message is higher because each door lock has synchronized the one-open-multiple-entry trajectory list.

[0077] (8)Send the one-open-multiple-entry event message to the platform in unicast mode: Send the one-open-multiple-entry events where all multiple incoming personnel are matched to faces to the platform according to the requirements of the platform.

[0078] There are 2 ways for the door lock to send the one-open-multiple-entry event to the platform: One is to send it quasi-real-time, and immediately send the one-open-multiple-entry event after real-time recognition and real-time matching; the other is to send it asynchronously. The one-open-multiple-entry event is first delayed and cached (set the delay length to 10 minutes), updated according to the possible cross-lock supplementary messages, and then sent to the platform.

[0079] Quasi-real-time sending: If A has recognized the face images / feature vectors of all personnel in a one-open-multiple-entrance event, it sends the one-open-multiple-entrance event message of A to the platform in unicast form. The data items include: event serial number, door lock ID, timestamp, unlocker number, unlocker's face image / feature vector, unlocker's structured features, face images / feature vectors of multiple accompanying persons, face matching confidence levels of multiple accompanying persons, and structured features of multiple accompanying persons. The platform receives the one-open-multiple-entrance event message sent by A and stores and applies it according to the platform's business process.

[0080] Asynchronous sending: If A has recognized the face images / feature vectors of all personnel in a one-open-multiple-entrance event and finds that there are data items with medium confidence levels among them, it sends the corrected one-open-multiple-entrance event information to the platform within a delay period of 10 minutes in total according to the processing result in step (5).

[0081] Note: After running for a period of time, the processing processes of A, B, and all other intelligent locks are the same. The processes of A and B are used as examples above mainly for the convenience of describing the processing processes of each process; there is a certain difference in real-time performance between quasi-real-time sending and asynchronous sending, but the recognition of one-open-multiple-entrance in asynchronous sending may be more accurate; after the platform receives the one-open-multiple-entrance event message of the intelligent lock, it can conduct population control, trajectory characterization, etc. The details of the application aspect are not the focus of the present invention.

[0082] It can be seen that within the capture time period, the face information and body information of multiple entering personnel are extracted; if a face image and a body image are simultaneously extracted on the same personnel image, the pairing is directly successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the one-open-multiple-entrance trajectory list, the similarity of the body information is calculated. If the similarity is greater than the preset threshold, the pairing is successful; for the body information that fails to be retrieved and paired successfully, its spatio-temporal trajectory sequence is constructed and compatibility calculation is performed with the spatio-temporal trajectory sequence of face clustering in the one-open-multiple-entrance trajectory list to determine the confidence level; the various pairing results are fused, and for the face information corresponding to each body image information, the one with the highest face matching confidence level is taken as the fusion result, so as to be able to construct an identifier of the result of multiple entering personnel with confidence and realize the recognition of multiple entering personnel.

[0083] Another embodiment of the present invention provides a face recognition system for one-open-multiple-entrance of an intelligent lock. Refer to Figure 3 and the system may include: A monitoring module 301 for detecting multiple entering personnel: within the capture time period, extracting the face information and body information of multiple entering personnel; The pairing module 302 is used for multi-mode pairing: direct pairing and exclusion pairing. If a face image and a body image are simultaneously extracted from the same person image, the direct pairing is successful, and the confidence level is at the definite level. If the unique pairing relationship is determined by the exclusion method, the pairing is successful, and the confidence level is at the definite level. Face retrieval pairing: For the body information that has not been directly paired or successfully paired by exclusion, by retrieving a one-open-multiple-entry trajectory list, calculate the similarity of the body information. If the similarity is greater than the preset threshold, the pairing is successful, and the confidence level is medium or high. Spatiotemporal trajectory compatibility pairing: For the body information that has not been successfully paired by retrieval, construct its spatiotemporal trajectory sequence, and perform compatibility calculation with the spatiotemporal trajectory sequence of the face clustering in the one-open-multiple-entry trajectory list to determine the confidence level. The fusion module 303 is used for fusing the pairing results: fuse each pairing result, and for the face information corresponding to each body image information, select the one with the highest face matching confidence level as the fusion result.

[0084] It can be seen that during the capture time period, the face information and body information of the multi-entry personnel are extracted. If a face image and a body image are simultaneously extracted from the same person image, the direct pairing is successful. If the unique pairing relationship is determined by the exclusion method, the pairing is successful. By retrieving a one-open-multiple-entry trajectory list, calculate the similarity of the body information. If the similarity is greater than the preset threshold, the pairing is successful. For the body information that has not been successfully paired by retrieval, construct its spatiotemporal trajectory sequence, and perform compatibility calculation with the spatiotemporal trajectory sequence of the face clustering in the one-open-multiple-entry trajectory list to determine the confidence level. Fuse each pairing result, and for the face information corresponding to each body image information, select the one with the highest face matching confidence level as the fusion result, so as to be able to construct an identification of the multi-entry personnel result with confidence and realize the identification of the multi-entry personnel.

[0085] The embodiment of the present invention also provides a storage medium, in which a computer program is stored, and wherein the computer program is set to execute the steps in any one of the above method embodiments when running.

[0086] Specifically, in this embodiment, the above storage medium can be set to store a computer program for executing the following steps: S201, multi-entry personnel detection: During the capture time period, extract the face information and body information of the multi-entry personnel; S202, Multi-mode pairing: direct pairing and exclusion pairing: If a face image and a human body image are simultaneously extracted from the same person image, the direct pairing is successful and the confidence level is at the definite level; If the unique pairing relationship is determined by the exclusion method, the pairing is successful and the confidence level is at the definite level; Face retrieval pairing: For the human body information that has not been successfully paired directly or by exclusion, by retrieving a one-open-multi-entry trajectory list, calculate the similarity of the human body information. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; Spatiotemporal trajectory compatibility pairing: For the human body information that has not been successfully paired by retrieval, construct its spatiotemporal trajectory sequence, and perform compatibility calculation with the spatiotemporal trajectory sequence of the face clustering in the one-open-multi-entry trajectory list to determine the confidence level; S203, Pairing result fusion: Integrate the respective pairing results. For the face information corresponding to each human body image information, select the one with the highest face matching confidence level as the fusion result.

[0087] It can be seen that during the capture time period, the face information and human body information of multiple entering persons are extracted; If a face image and a human body image are simultaneously extracted from the same person image, the direct pairing is successful; If the unique pairing relationship is determined by the exclusion method, the pairing is successful; By retrieving a one-open-multi-entry trajectory list, calculate the similarity of the human body information. If the similarity is greater than the preset threshold, the pairing is successful; For the human body information that has not been successfully paired by retrieval, construct its spatiotemporal trajectory sequence, and perform compatibility calculation with the spatiotemporal trajectory sequence of the face clustering in the one-open-multi-entry trajectory list to determine the confidence level; Integrate the respective pairing results. For the face information corresponding to each human body image information, select the one with the highest face matching confidence level as the fusion result, so as to be able to construct an identification of the results of multiple entering persons with confidence and achieve the identification of multiple entering persons.

[0088] The embodiment of the present invention also provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0089] Specifically, the above electronic device may further include a transmission device and an input / output device. Among them, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0090] Specifically, in this embodiment, the above processor may be configured to execute the following steps through a computer program: S201, Detection of multiple entering persons: During the capture time period, extract the face information and human body information of multiple entering persons; S202, Multi - mode pairing: direct pairing and exclusion pairing: If a face image and a body image are simultaneously extracted from the same person image, the direct pairing is successful, and the confidence level is at the definite level; if the unique pairing relationship is determined by the exclusion method, the pairing is successful, and the confidence level is at the definite level; face retrieval pairing: For the body information that has not been directly paired or successfully paired by exclusion, by retrieving a one - in - multiple - out trajectory list, calculate the similarity of the body information. If the similarity is greater than the preset threshold, the pairing is successful, and the confidence level is medium or high; spatio - temporal trajectory compatibility pairing: For the body information that has not been successfully paired by retrieval, construct its spatio - temporal trajectory sequence, and perform compatibility calculation with the spatio - temporal trajectory sequence of the face clustering in the one - in - multiple - out trajectory list to determine the confidence level. S203, Pairing result fusion: Integrate each pairing result. For the face information corresponding to each body image information, select the one with the highest face matching confidence level as the fusion result.

[0091] It can be seen that during the capture time period, the face information and body information of multiple - entry personnel are extracted; if a face image and a body image are simultaneously extracted from the same person image, the direct pairing is successful; if the unique pairing relationship is determined by the exclusion method, the pairing is successful; by retrieving a one - in - multiple - out trajectory list, calculate the similarity of the body information. If the similarity is greater than the preset threshold, the pairing is successful; for the body information that has not been successfully paired by retrieval, construct its spatio - temporal trajectory sequence, and perform compatibility calculation with the spatio - temporal trajectory sequence of the face clustering in the one - in - multiple - out trajectory list to determine the confidence level; integrate each pairing result. For the face information corresponding to each body image information, select the one with the highest face matching confidence level as the fusion result, so as to be able to construct an identification of the result of multiple - entry personnel with confidence and achieve the identification of multiple - entry personnel.

[0092] The above has described in detail the structure, features and function effects of the present invention according to the illustrated embodiments. The above is only the preferred embodiment of the present invention, but the present invention is not limited to the scope defined by the drawings. Any changes made according to the concept of the present invention, or equivalent embodiments modified into equivalent changes, still within the spirit covered by the specification and the drawings, should be within the protection scope of the present invention.

Claims

1. A face recognition method for a smart lock with multiple entries at once, characterized in that: The method comprises: Personnel detection: extract facial and body information of multiple persons entering during the capture period; Multi-mode pairing: direct pairing and exclusion pairing: if the face image and body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is definite; if the unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is definite; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by retrieving the one-open-multiple-entry trajectory list. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not retrieved and paired successfully, its spatiotemporal trajectory sequence is constructed, and the compatibility is calculated with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence level; Pairing result fusion: fuse the various pairing results, and for each face information corresponding to the human image information, take the face with the highest confidence level as the fusion result.

2. The method according to claim 1, characterized in that The direct pairing includes: Based on the video images captured by the smart lock, the face detection algorithm is used to extract the face image / feature vector, and the body detection algorithm is used to extract the body image / structural features; If the face image / feature vector and the body image / structural feature are extracted from the same person image at the same time, it is directly determined that the pairing is successful, and the confidence level is confirmed; If the direct matching method is not completely successful, the exclusion matching method is used to uniquely match the remaining face images / feature vectors and body images / structured features. If the remaining face images / feature vectors and body images / structured features are the only matching relationship, the matching is successful and the confidence level is certain.

3. The method according to claim 2, characterized in that The face retrieval pairing includes: For human images / structured features that are not directly matched successfully, the similarity between them and the human images / structured features in the list is calculated by retrieving the one-open-multiple-entry trajectory list stored in the smart lock; If the similarity is greater than the preset threshold, the corresponding face image / feature vector is extracted from the one-open-multiple-input trajectory list, and the pairing is successful with a medium or high confidence level; Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

4. The method according to claim 3, characterized in that The space-time trajectory compatibility pairing includes: According to the unpaired human body image / structural features, construct its spatiotemporal trajectory sequence, including time trajectory sequence and space trajectory sequence; Compare the constructed spatiotemporal trajectory sequence with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-input trajectory list to determine whether they are completely overlapped, partially overlapped, or completely compatible; If the space-time trajectories completely overlap, the confidence level is certain; if they partially overlap and are completely compatible, the confidence level is high; if they do not overlap but are completely compatible, the confidence level is medium; Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

5. The method according to claim 4, characterized in that The pairing result fusion includes: The pairing results are merged to obtain the face image / feature vector with the highest confidence level corresponding to each human image / structural feature; Generate complete one-open-multiple-entry event information, the one-open-multiple-entry event information includes event sequence number, door lock ID, time, unlocker number, multiple companion face images / feature vectors, multiple companion face matching confidences, and multiple companion structured features; Send the one-open-multiple-entry event information to other smart locks in multicast form, and send it to the platform in unicast form; The one-time multiple-entry event information is written into the one-time multiple-entry track list of the smart lock. The storage time is 10 minutes and it will be automatically deleted after the expiration.

6. A face recognition system for smart locks with multiple entries, characterized in that: The system comprises: Monitoring module, used for multiple-person detection: extracting facial and body information of multiple people during the capture period; Pairing module, used for multi-mode pairing: direct pairing and exclusion pairing: if the face image and body image are extracted from the same person image at the same time, the direct pairing is successful, and the confidence level is definite; if the unique pairing relationship is determined by the elimination method, the pairing is successful, and the confidence level is definite; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by retrieving the one-open-multiple-input trajectory list. If the similarity is greater than the preset threshold, the pairing is successful, and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not retrieved successfully, its spatiotemporal trajectory sequence is constructed, and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-input trajectory list to determine the confidence level; The fusion module is used to fuse the pairing results: each pairing result is fused, and for each face information corresponding to the human image information, the face with the highest confidence level is taken as the fusion result.

7. The system according to claim 6, characterized in that The pairing module is specifically used for: Based on the video images captured by the smart lock, the face detection algorithm is used to extract the face image / feature vector, and the body detection algorithm is used to extract the body image / structural features; If the face image / feature vector and the body image / structural feature are extracted from the same person image at the same time, it is directly determined that the pairing is successful, and the confidence level is confirmed; If the direct matching method is not completely successful, the exclusion matching method is used to uniquely match the remaining face images / feature vectors and body images / structured features. If the remaining face images / feature vectors and body images / structured features are the only matching relationship, the matching is successful and the confidence level is certain.

8. The system according to claim 7, characterized in that The pairing module is specifically used for: For human images / structured features that are not directly matched successfully, the similarity between them and the human images / structured features in the list is calculated by retrieving the one-open-multiple-entry trajectory list stored in the smart lock; If the similarity is greater than the preset threshold, the corresponding face image / feature vector is extracted from the one-open-multiple-input trajectory list, and the pairing is successful with a medium or high confidence level; Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

9. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 5 when executed.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 5.

Images