A face recognition method and system for smart locks with multiple entries

Abstract

The present invention discloses a face recognition method and system for a smart lock with multiple entries. The method includes: extracting facial information and body information of multiple entries within a capture time period; if facial images and body images are simultaneously extracted from the same person image, direct pairing is successful; if a unique pairing relationship is determined by the elimination method, pairing is successful; by searching a list of trajectories for multiple entries, calculating the similarity of the body information, and if the similarity is greater than a preset threshold, pairing is successful; for the body information that is not successfully retrieved, constructing its spatiotemporal trajectory sequence, and performing compatibility calculation with the face clustering spatiotemporal trajectory sequence in the list of trajectories for multiple entries to determine the confidence level; fusing the various pairing results, and for each face information corresponding to the body image information, taking the face matching result with the highest confidence level as the fusion result. Utilizing the embodiments of the present invention, it is possible to construct a confident identification of the results of multiple entries, thereby realizing the recognition of multiple entries.

Description

Technical Field

[0001] The present invention belongs to the technical field of smart locks, and in particular to a face recognition method and system for a smart lock with one-click multi-entry. Background Art

[0002] With the acceleration of urbanization, the number of migrant population in cities is increasing, and rental houses and apartments are also undergoing a transformation. Public security agencies in many places are also promoting the installation of smart locks in rental houses / apartments. Smart locks are gradually entering the management of rental houses / apartments and residential houses, bringing many conveniences to the management agencies of rental houses / apartments and local public security management departments, and improving the safety and orderliness of rental houses / apartments.

[0003] Smart locks, as security facilities, provide residents with a sense of security while also providing management agencies with records of residents entering and exiting. Currently, the technologies used by smart locks to identify people entering and exiting primarily rely on fingerprint and facial recognition. To fully utilize video images to monitor the conditions at the door, smart locks installed at the entrance are equipped with one or more cameras. The video images of people captured on-site are shared with other smart locks and can also be aggregated to the smart lock management platform. However, currently, smart locks can only accurately identify the person who unlocked the door. People entering and exiting with the unlocking person cannot be effectively identified due to non-cooperation with unlocking recognition, the camera angle of the door lock, the person's position, obstructions, and other factors, creating a blind spot for security management. Summary of the Invention

[0004] The purpose of the present invention is to provide a face recognition method and system for a smart lock with multiple entries, so as to solve the deficiencies in the prior art, and to construct a confident identification of the results of multiple entries, thereby realizing the identification of multiple entries.

[0005] One embodiment of the present application provides a face recognition method for a smart lock with multiple entries, the method comprising:

[0006] Multiple-entry person detection: Extract facial and body information of multiple people within the capture period;

[0007] Multi-modal pairing: direct pairing and exclusion pairing: if the face image and body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if the unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by retrieving the one-open multiple-entry trajectory list. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not retrieved and paired successfully, its spatiotemporal trajectory sequence is constructed and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the one-open multiple-entry trajectory list to determine the confidence level;

[0008] Pairing result fusion: fuse all pairing results, and for each face information corresponding to the human image information, take the face matching with the highest confidence as the fusion result.

[0009] Optionally, the direct pairing includes:

[0010] Based on the video images captured by the smart lock, the face detection algorithm is used to extract the face image / feature vector, and the human body detection algorithm is used to extract the human body image / structural features;

[0011] If both the facial image / feature vector and the body image / structural features are extracted from the same person image, the pairing is directly determined to be successful, and the confidence level is confirmed;

[0012] If the direct matching method is not completely successful, the exclusion matching method is used to uniquely match the remaining facial images / feature vectors and body images / structured features. If the remaining facial images / feature vectors and body images / structured features are the only matching relationship, the matching is successful and the confidence level is certain.

[0013] Optionally, the face retrieval pairing includes:

[0014] For human images / structured features that are not directly matched successfully, the similarity between them and the human images / structured features in the list of one-open-multiple-entry trajectories stored in the smart lock is calculated;

[0015] If the similarity is greater than the preset threshold (80%), the corresponding face image / feature vector is extracted from the one-open-multiple-input trajectory list, and the pairing is successful with a medium or high confidence level;

[0016] Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

[0017] Optionally, the space-time trajectory compatibility pairing includes:

[0018] According to the unpaired human body image / structural features, its spatiotemporal trajectory sequence is constructed, including the time trajectory sequence and the spatial trajectory sequence;

[0019] Compare the constructed spatiotemporal trajectory sequence with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-input trajectory list to determine whether they are completely overlapping, partially overlapping, or completely compatible;

[0020] If the space-time trajectories completely overlap, the confidence level is certain; if they partially overlap and are completely compatible, the confidence level is high; if they do not overlap but are completely compatible, the confidence level is medium;

[0021] Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

[0022] Optionally, the pairing result fusion includes:

[0023] Fuse each pairing result and take the face image / feature vector with the highest confidence corresponding to each human image / structural feature;

[0024] Generate complete one-open-multiple-entry event information, including the event sequence number, door lock ID, time, unlocker ID, multiple companion face images / feature vectors, multiple companion face matching confidences, and multiple companion structured features;

[0025] Send the one-open-multiple-entry event information to other smart locks in multicast form and to the platform in unicast form;

[0026] The one-time multiple-entry event information is written into the one-time multiple-entry track list of the smart lock. The storage time is 10 minutes and it will be automatically deleted after the expiration.

[0027] Another embodiment of the present application provides a face recognition system for a smart lock with multiple entries, the system comprising:

[0028] Monitoring module, used for multiple-entry person detection: extracting facial and body information of multiple people within the capture period;

[0029] The pairing module is used for multi-modal pairing: direct pairing and exclusion pairing: if the face image and body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if the unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by retrieving the one-open multiple-entry trajectory list. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not retrieved and paired successfully, its spatiotemporal trajectory sequence is constructed and the compatibility calculation is performed with the face cluster spatiotemporal trajectory sequence in the one-open multiple-entry trajectory list to determine the confidence level;

[0030] The fusion module is used to fuse the pairing results: each pairing result is fused, and for each face information corresponding to the human image information, the face with the highest confidence level is taken as the fusion result.

[0031] Yet another embodiment of the present application provides a storage medium, wherein the storage medium stores a computer program, wherein the computer program is configured to execute any of the above methods when run.

[0032] Yet another embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute any of the above methods.

[0033] Compared with the prior art, the present invention provides a face recognition method for multiple entries in a smart lock. Within the capture time period, the face information and body information of multiple entries are extracted; if the face image and body image are extracted simultaneously on the same person image, the direct pairing is successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the single-entry multiple-entry trajectory list, the similarity of the body information is calculated, and if the similarity is greater than a preset threshold, the pairing is successful; for the body information that is not retrieved and paired successfully, its spatiotemporal trajectory sequence is constructed, and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the single-entry multiple-entry trajectory list to determine the confidence level; the various pairing results are fused, and for each face information corresponding to the body image information, the face matching result with the highest confidence level is taken as the fusion result, thereby being able to construct a confident identification of the results of multiple entries and realize the recognition of multiple entries. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 A hardware structure block diagram of a computer terminal for a face recognition method for a smart lock with multiple entries provided by an embodiment of the present invention;

[0035] Figure 2A flowchart of a face recognition method for a smart lock with multiple entries provided by an embodiment of the present invention;

[0036] Figure 3 A structural diagram of a face recognition system for a smart lock with one-click multi-entry provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0037] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and are not to be construed as limiting the present invention.

[0038] The embodiment of the present invention first provides a face recognition method for a smart lock with one-time opening and multiple entry. The method can be applied to electronic devices such as computer terminals, specifically ordinary computers, etc.

[0039] The following describes it in detail by taking running on a computer terminal as an example. Figure 1 The hardware structure block diagram of the computer terminal of the face recognition method for a smart lock with multiple entries provided by the embodiment of the present invention. Figure 1 As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus, wherein the memory may include a non-volatile storage medium and an internal memory.

[0040] The non-volatile storage medium can store an operating system and a computer program. The computer program includes program instructions that, when executed, enable the processor to perform any of the face recognition methods for smart locks with multiple entries.

[0041] The processor is used to provide computing and control capabilities and support the operation of the entire computer equipment.

[0042] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When the computer program is executed by the processor, the processor can execute any face recognition method for multiple entries in a smart lock.

[0043] The network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art will understand that Figure 1 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0044] It should be understood that the processor may be a central processing unit (CPU), other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor or any conventional processor, etc.

[0045] See also Figure 2 The embodiment of the present invention provides a face recognition method for a smart lock with multiple entries, which may include the following steps:

[0046] S201, multiple-entry person detection: extracting facial information and body information of multiple-entry persons within the capture time period;

[0047] S202, multi-mode pairing, including:

[0048] Direct matching and exclusion matching: If both the face image and the body image are extracted from the same person image, the direct matching is successful and the confidence level is certain. If a unique matching relationship is determined through the elimination method, the matching is successful and the confidence level is certain. Specifically, the direct matching includes:

[0049] Based on the video images captured by the smart lock, the face detection algorithm is used to extract the face image / feature vector, and the human body detection algorithm is used to extract the human body image / structural features;

[0050] If both the facial image / feature vector and the body image / structural features are extracted from the same person image, the pairing is directly determined to be successful, and the confidence level is confirmed;

[0051] If the direct matching method is not completely successful, the exclusion matching method is used to uniquely match the remaining facial images / feature vectors and body images / structured features. If the remaining facial images / feature vectors and body images / structured features are the only matching relationship, the matching is successful and the confidence level is certain.

[0052] Face retrieval and matching: For human body information that is not directly matched or excluded from successful matching, the similarity of the human body information is calculated by searching the single-open multiple-entry trajectory list. If the similarity is greater than a preset threshold, the matching is successful and the confidence level is medium or high. Specifically, the face retrieval and matching includes:

[0053] For human images / structured features that are not directly matched successfully, the similarity between them and the human images / structured features in the list of one-open-multiple-entry trajectories stored in the smart lock is calculated;

[0054] If the similarity is greater than the preset threshold (80%), the corresponding face image / feature vector is extracted from the one-open-multiple-input trajectory list, and the pairing is successful with a medium or high confidence level;

[0055] Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

[0056] Spatiotemporal trajectory compatibility pairing: For human body information that has not been successfully retrieved and matched, its spatiotemporal trajectory sequence is constructed and its compatibility is calculated with the face cluster spatiotemporal trajectory sequence in the one-open-multiple-input trajectory list to determine the confidence level. Specifically, the spatiotemporal trajectory compatibility pairing includes:

[0057] According to the unpaired human body image / structural features, its spatiotemporal trajectory sequence is constructed, including the time trajectory sequence and the spatial trajectory sequence;

[0058] Compare the constructed spatiotemporal trajectory sequence with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-input trajectory list to determine whether they are completely overlapping, partially overlapping, or completely compatible;

[0059] If the space-time trajectories completely overlap, the confidence level is certain; if they partially overlap and are completely compatible, the confidence level is high; if they do not overlap but are completely compatible, the confidence level is medium;

[0060] Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

[0061] S203, pairing result fusion: fuse the pairing results, and for each face information corresponding to the human image information, take the face matching with the highest confidence as the fusion result. Specifically, the pairing result fusion includes:

[0062] Fuse each pairing result and take the face image / feature vector with the highest confidence corresponding to each human image / structural feature;

[0063] Generate complete one-open-multiple-entry event information, including the event sequence number, door lock ID, time, unlocker ID, multiple companion face images / feature vectors, multiple companion face matching confidences, and multiple companion structured features;

[0064] Send the one-open-multiple-entry event information to other smart locks in multicast form and to the platform in unicast form;

[0065] The one-time multiple-entry event information is written into the one-time multiple-entry track list of the smart lock. The storage time is 10 minutes and it will be automatically deleted after the expiration.

[0066] Smart door locks, also known as intelligent locks, are improved upon traditional mechanical locks, offering enhanced intelligence and simplicity in terms of user security, identification, and management. The smart door locks mentioned in this article have facial recognition capabilities for unlocking the door.

[0067] Currently, facial target clustering is commonly used. For example, T / CSPIA 008-2022, "Technical Requirements for Video Image Target Clustering Services," mentions clustering and archiving video target objects collected from a wide range of video images to build a video image target archive. Clustering can be performed in two main ways: regular full clustering and incremental clustering each time a target is added. This clustering method requires the construction of a very large GPU computing cluster to perform clustering calculations on massive amounts of data. If used for the identification of multiple people entering a smart lock, clustering cannot be achieved because the information reported by each smart lock does not provide information about the communicating person.

[0068] The purpose of the present invention is to provide a method for identifying the faces of multiple-entry personnel whose faces have not been recognized in a one-open-multiple-entry mode, and to share the one-open-multiple-entry recognition trajectory points between smart locks by multicasting, so as to provide complete face recognition of all personnel in the one-open-multiple-entry mode in the message reported by the smart lock to the platform. Each smart lock stores the one-open-multiple-entry information of the door lock detection and recognition shared by each smart lock through multicasting, and performs face matching of multiple-entry personnel whose faces have not been recognized in a one-open-multiple-entry mode by combining the compatibility of spatiotemporal trajectories and the structural features of human images, and uses three levels of confidence to identify the credibility of the face matching, which provides a reference for the pairing of other door locks and also provides support for platform applications. In actual applications, a complete process of a technical implementation is as follows:

[0069] (1) Prerequisites:

[0070] Smart locks are installed on residents' entrance doors (each door lock has an independent ID, door lock IP address, door lock geographic location, configured multicast group IP address, etc.), and are connected to the smart lock management platform (hereinafter referred to as the platform) and registered with the platform. They can receive the remote configuration multicast group IP address of the platform's network management tool and periodically update the occupant registration list from the platform (data items include: occupant number, name, ID number, facial image / feature vector); smart locks can be connected to other smart locks through multicast.

[0071] The platform records a list of smart locks, including door lock ID, door lock IP address, door lock geographic location, multicast group, and multicast group IP address; the platform configures the smart locks of a building or unit into a multicast group based on the building or unit situation in the community. Smart door locks: also known as smart locks, refer to locks that are improved based on traditional mechanical locks and are more intelligent and simple in terms of user security, identification, and management. The smart door locks referred to in this article have the function of opening the door through facial recognition. A few concepts are explained:

[0072] 1) Multiple entries at once refers to the situation where a resident who registers to check in unlocks the door and multiple people enter together. Due to the installation position of the door lock camera and the obstruction between multiple people, it is impossible to capture the facial images of all people, resulting in the inability to identify all the people entering at once.

[0073] 2) Multiple entrants refer to those who enter together with the resident registration person after unlocking the door. If the resident registration person unlocks the door and enters alone, it can be considered a special case of multiple entrants.

[0074] 3) Capture time period: This refers to the time period from when multiple people enter the door through the camera shooting area to when all of them enter the door for capture and identification.

[0075] 4) Single-open multiple-entry trajectory list: The smart lock stores information about single-open multiple-entry events collected by the lock itself or sent by other locks through multicast, including event sequence number, door lock ID, time, unlocker number, multiple facial images / feature vectors of the unlocked person, multiple facial matching confidences of the unlocked person, and multiple structural features of the unlocked person.

[0076] (2) Face matching confidence rules: set three confidence levels according to different situations

[0077] The confidence level in this article indicates the likelihood that a facial image / feature vector extracted from a video image and a body image / structural feature are the same person. A higher confidence level indicates a higher likelihood that the facial image and body image are the same person. This is primarily intended to indicate the confidence that a person who only captured the body image can identify the facial image. Confidence levels are categorized into three levels: certain, high, and medium.

[0078] Deterministic confidence: When the smart lock captures both a face and a body image, direct pairing is successful, or at least one of the received trajectory points has a confidence level for the body image (structured features) of the deterministic level, or complete overlap is found through comparison of the spatiotemporal trajectory sequences.

[0079] High confidence: If the spatiotemporal trajectory sequences partially overlap and are completely compatible; or if the spatiotemporal trajectory sequences do not overlap but are completely compatible, and the person corresponding to the face trajectory is the unlocked person.

[0080] Medium confidence: or the spatiotemporal trajectory sequences do not overlap, but are completely compatible.

[0081] Face image / feature vector: refers to the image of the face area extracted by the smart lock from the video image captured by the smart lock camera based on the face intelligent detection, recognition and segmentation algorithm, and the feature vector of the face image is extracted.

[0082] Human image / structured features: This refers to the image of the person area extracted from the video image captured by the smart lock camera using the intelligent person detection and segmentation algorithm. The structured features of the human image are then extracted using a structured parsing algorithm. The human body here refers to the entire person; the image quality of the face does not meet the requirements for facial recognition.

[0083] A spatiotemporal trajectory point refers to the current time (i.e., time location) when capturing and extracting facial images / feature vectors, body images / structured features, and the smart lock's geographic location (i.e., spatial location, represented by the smart lock ID number, which is clearly associated with a geographic location). Spatiotemporal trajectory points can be sorted by time to form a temporal trajectory sequence, and by geographic location to form a spatial trajectory sequence. In this article, a spatiotemporal trajectory point refers to a trajectory point collected during the start and end of a single-open, multiple-entry event.

[0084] Trajectory point sequence compatibility refers to the situation where two uncontrolled trajectory point sequences have no overlapping trajectory points in either temporal or spatial trajectory sequences, but do exist in the intervals between them. For example, suppose door locks A-Z are sorted by spatial order (route), and time points A-Z are sorted by temporal order. A certain structured information is captured by (door lock A, time point A), (door lock C, time point C), and (door lock E, time point E). A certain face is captured by (door lock B, time point B), (door lock D, time point D), and (door lock F, time point F). Although the structured information and the face do not share common spatiotemporal overlap, they are clearly compatible and have a high probability of consistency.

[0085] (3) Perform face matching of multiple people entering the door during the door lock capture period: The smart lock uses video images to intelligently track and identify the faces and body images of multiple people entering the door, and then matches the faces and bodies;

[0086] Detection and recognition of faces and bodies of multiple people: During the smart door lock capture period, facial images (feature vectors are extracted through intelligent analysis) and body images (structured features are extracted after structured analysis) are extracted through intelligent detection and recognition technology.

[0087] For example: when smart lock A detects and identifies multiple people entering at the same time, the check-in person a of resident A undergoes facial recognition comparison and passes A's comparison and unlocks the door. At this time, other people (such as b, c, d) enter the door with a. However, due to the installation position and angle of the smart lock's camera, and obstructions between people, only the faces of b and c can be captured, and the face of d is not captured. However, the body images of all people (a, b, c, d) can be collected.

[0088] There are two methods for pairing facial images / feature vectors and body images / structured features of multiple people entering the capture time period: direct pairing and exclusion pairing.

[0089] Direct matching method: If the face image / feature vector and body image / structured features are extracted from the same person image at the same time, the matching is determined to be successful and the confidence level is determined (determined according to the confidence determination rule in step (2)).

[0090] For example: If the smart lock captures a full-face image of a person, including both the face and the complete appearance of the person, the facial image / feature vector and body image / structured features can be extracted separately. Moreover, since they are extracted from the image of the same person, it is obviously the same person.

[0091] Elimination matching: In a single-input, multiple-input extraction of multiple facial images / feature vectors, body images, and structured features, if one pair of facial images / feature vectors, body images, and structured features is successfully matched with two people, the remaining pair of facial images / feature vectors, body images, and structured features can be directly matched with the other person. Similarly, in a multi-person case, if only one pair of facial images / feature vectors, body images, and structured features remains after direct matching, it can be directly matched with the other person. If multiple facial images / feature vectors, body images, and structured features remain, direct matching cannot be performed. The confidence level for successful matching using the elimination matching method is determined to be certain.

[0092] For example: Let’s take the case of 2 people and the case of multiple people as examples.

[0093] In the case of two people: If the facial images / feature vectors collected during the one-open multiple-input time period are (F1, F2), and the body images / structured features are (V1, V2), and if it is clear that F1 and V2 are successfully paired using the direct pairing method, then the remaining F2 and V1 can be paired successfully. Since there are only two people, the remaining one can only be the second person;

[0094] If there are multiple people: Take 3 people as an example. If the facial images / feature vectors collected in the one-open-multiple-entry event segment are (F1, F2, F3), and the body images / structured features are (V1, V2, V3), and if it is clear that F1 and V2, and F2 and V1 are successfully paired directly, then the remaining F3 and V3 can be paired successfully. Since there are only 3 people, the remaining person can only be the third person. By analogy, as long as there is only one person left and only one pair of facial images / feature vectors and body images / structured features, the remaining pair of facial images / feature vectors and body images / structured features can be paired successfully.

[0095] If the confidence level of all multi-entry persons successfully matching to the face is certain within the one-open-multiple-entry time period, then skip steps (4) and (5), and directly write the one-open-multiple-entry trajectory list of the door lock, and send it to other door locks in multicast form for synchronization of the one-open-multiple-entry trajectory list, and send it to the platform in unicast form for platform application.

[0096] (4) Perform multi-entry face retrieval and matching in the door lock storage record: Use the human body image / structural features that were not successfully matched to the face image in step (3) to compare and retrieve the human body image / structural features in the one-open-multiple-entry trajectory list to match the face image / feature vector;

[0097] The search is performed in a list of multiple tracks. Based on the similarity of body images / structural features, the face image / feature vector corresponding to the largest body image / structural feature in the list of multiple tracks with a similarity value greater than a threshold (generally set to 80%) is used, along with the confidence level of the face match. If all similarity values are less than the threshold, the search fails.

[0098] Note: The smart door lock's single-entry, multiple-entry track list synchronously stores the single-entry, multiple-entry event information for each member of the multicast group. Before the captured, multiple-entry event information is written to the lock's single-entry, multiple-entry track list, a multicast message is sent, and a message is sent to the platform, the faces of all multiple entrants must be identified (possibly with low confidence levels). If the similarity values are all below the threshold, the similarity between the human image / structured features is low, and the likelihood of them being the same person is low. Matching the facial images at this time will result in significant errors.

[0099] (5) Compatibility matching of the spatiotemporal trajectories of multiple people in the door lock storage records: Compatibility calculation is performed on the spatiotemporal trajectory sequence of multiple people constructed with human body images / structured features and the spatiotemporal trajectory sequence after face clustering, and pairing is performed according to the face matching confidence rule and the face matching confidence is determined.

[0100] Face clustering and spatiotemporal trajectory sequence construction: Cluster calculation is performed based on all face images / feature vectors in the one-open-multiple-entry trajectory list. The clustered trajectory points (note: trajectory points are records of one-open-multiple-entry events with spatiotemporal characteristics) are constructed into a spatiotemporal trajectory sequence according to their temporal sequence (timestamp) and spatial sequence (door lock ID number).

[0101] For example: The clustering algorithm for video images calculates the similarity between facial images / feature vectors, and files those that meet the similarity threshold into a class corresponding to a person; if the faces in the face image / feature vector list stored in the door lock are a, b, c, d, ..., z, among which face a is clustered into a1, a2, a3, ...an, with a total of n trajectory points (face b, ..., z and so on), the time trajectory sequence T1, T2, ...Tn is constructed according to the timestamp, and the spatial trajectory sequence W1, W2, ...Wn is constructed according to the order of the door lock ID number.

[0102] Human body image / structured feature retrieval and spatiotemporal trajectory sequence construction for multiple entries: The human body image / structured feature retrieval of multiple entries is used to compare all human body images / structured features in the open multi-entry trajectory list, extract all trajectory points with a similarity ≥ a threshold (generally set to 80%), and construct a spatiotemporal trajectory sequence according to the temporal order (timestamp) and spatial order (door lock ID number).

[0103] For example, if the human image / structural feature of the multi-entry person is x, after similarity calculation with all human images / structural features in the one-open-multi-entry trajectory list, all trajectory points S1, S2, ..., Sm with similarity ≥ 80% are extracted. These m trajectory points are constructed into a time trajectory sequence t1, t2, ... tm according to the timestamp, and a spatial trajectory sequence w1, w2, ... wm is constructed according to the order of the door lock ID number.

[0104] Compatibility of spatiotemporal trajectory sequences: The temporal and spatial trajectory sequences of multiple individuals are compared point by point with the temporal and spatial trajectory sequences of the face clusters to check for consistency in time, space, and order. If the temporal and spatial trajectory sequences are consistent, the trajectories overlap. If they do not overlap, but fall in the middle of the other spatiotemporal trajectory sequence, the trajectories are compatible.

[0105] Description of spatiotemporal trajectory sequence: If the time trajectory sequence of the human body image / structural feature x of multiple people is t1, t2, ..., tm, and the spatial trajectory sequence is w1, w2, ..., wm, compare t1, t2, ..., tm with the time trajectory sequence T1, T2, ...Tn of face a in face clustering one by one. If t1=T1, t2=T2, ..., tm=Tm, it means that every time point coincides, indicating that the time trajectory sequences coincide. Similarly, comparison can also be made for spatial sequences.

[0106] An example of trajectory sequence compatibility: If the time trajectory sequence of the human body image / structured feature x of multiple people is t1, t3, t5, t7, and the time trajectory sequence of face a is T2, T4, T6, if t1≤T2≤t3, t3≤T4≤t5, t5≤T6≤t7, then the time trajectory sequence of x is compatible with the time trajectory sequence of a; compatibility also includes partial compatibility and full compatibility. Full compatibility means that the time trajectory sequence of x and the time trajectory sequence of a can be completely inserted into each other's time slot, and partial compatibility means that part of the time trajectory sequence can be inserted into the time slot of the other.

[0107] The compatibility of spatiotemporal trajectory sequences determines the confidence of face matching: it is divided into several cases, see Table 1 below.

[0108] Table 1

[0109]

[0110] Description of determining face matching confidence based on spatiotemporal trajectory sequence compatibility:

[0111] 1) Complete spatiotemporal trajectory overlap: that is, the spatiotemporal trajectory sequence of multiple people completely overlaps with the spatiotemporal trajectory sequence after a certain face clustering, and the confidence level is certain;

[0112] Note: This situation usually occurs when a person's face and body images are captured in front of multiple door locks.

[0113] 2) The spatiotemporal trajectories partially overlap and are completely compatible: that is, the spatiotemporal trajectory sequences of multiple people overlap with the spatiotemporal trajectory sequence after a face clustering, and all other trajectory points are compatible. In this case, the confidence level is high.

[0114] Note: This situation usually occurs when a person's face and body images are captured simultaneously in front of multiple door locks, while in some cases only the face or body image is captured.

[0115] 3) The spatiotemporal trajectories do not overlap, but are completely compatible, and the face trajectory sequence is the unlocked person: If they are completely compatible, and the face in the spatiotemporal trajectory is a certain unlocked person (i.e., the check-in person of a certain resident), the confidence level is high.

[0116] Note: This situation usually occurs when a person is in front of multiple door locks. Some of them only capture the person's face or body image, but it can be clearly seen that someone directly unlocked the door, which may be a neighbor visiting.

[0117] 4) The space-time trajectories do not overlap, but are completely compatible: If the space-time trajectories are completely compatible, the confidence level is medium.

[0118] Note: This situation usually occurs when a person is in front of multiple door locks. Some of the images only capture the person's face or body. It is likely that it is the same person.

[0119] 5) The space-time trajectories do not overlap, are partially compatible or incompatible: only a few trajectory points are compatible, or even no compatible trajectory points. In this case, the confidence level is zero, i.e., no confidence;

[0120] Note: This situation usually occurs when the person does not appear in front of other door locks or is dressed somewhat similarly, but is not the same person. In this case, trust based on compatibility is meaningless.

[0121] The cross-lock supplement method is used to further correct the pairing results: smart lock A (assuming that A has multiple entries with one door open and multiple entries but has not detected and recognized their faces, and that the confidence level of the pairing has not reached a high confidence level through query retrieval or compatibility calculation of the single-open, multiple-entry trajectory list, in order to further improve the matching to high-confidence faces, a control method can be adopted to wait for other door locks to have higher face matching confidence levels for the person) waits for no more than 10 minutes, and when receiving the faces and confidence levels of the undetected multiple entries from B, compares the confidence level calculated previously by A with the confidence level just received from B, and takes the face with the higher confidence level as the final face and confidence level of the undetected multiple entries.

[0122] For example, suppose A has a multi-entry event with sequence number XXX01, which involves four people (a, b, c, and d). Person B is a multiple-entry person whose face was not captured. Using the trajectory compatibility method based on B's structured features, A calculates face F1 (F1's facial image / feature vector is extracted from A's stored multi-entry trajectory list, and the calculated confidence is medium). Within a 10-minute retention period, A receives multicast message YYY01 from B. B also recognizes B's presence and calculates the confidence level of F2 for B's matching with high confidence. A then discovers that B has provided a more confident face image / feature vector F2 for B. A then updates F2 to its own multi-entry trajectory list and updates the face image / feature vector of B in the multi-entry event XXX01 to F2.

[0123] (6) Fusion and processing of face matching results for one-open multiple-entry events: The smart door lock fuses the face matching results completed in steps (3), (4), and (5), and takes the face image / feature vector corresponding to each human image / structured feature as the fusion result with the highest face matching confidence.

[0124] The smart door lock performs fusion based on the face matching confidence: for each person entering multiple places, the face matching results completed in step (3), step (4), and step (5) are compared, and the corresponding face image / feature vector and face matching confidence with the highest face matching confidence are taken as the fusion result.

[0125] For example: if door lock A captures four persons, a, b, c, and d, a is the unlocker, and b, c, and d are traveling companions, i.e., multiple persons; if b is not directly captured with a facial image and a body image, in step (3), face matching of multiple persons is performed within the door lock capture time period, and the face matching confidence of the face image / feature vector x is determined; in step (4), face retrieval matching of multiple persons is performed in the door lock storage record, and the face matching confidence of the face image / feature vector y is medium; in step (5), spatiotemporal trajectory compatibility matching of multiple persons is performed in the door lock storage record, and the face matching confidence of the face image / feature vector z is medium. Then, the matching face image / feature vector for b should be determined as x, and its face matching confidence is determined.

[0126] After the smart door lock completes face matching, it creates a complete "one-time multiple-entry" event and writes it into the lock's "one-time multiple-entry" trajectory list. This data is then multicasted to other door locks and platforms. This data includes the event sequence number, lock ID, time, unlocker ID, multiple facial images / feature vectors, multiple facial match confidences, and multiple structural features. The data in the "one-time multiple-entry" trajectory list is retained for 10 minutes.

[0127] An example of a data item for a "one-open, multiple-entry" event is as follows: Door lock A captures four people, a, b, c, and d. A is the person who unlocked the door, and b, c, and d are traveling together, meaning there are multiple people entering the door.

[0128] Event sequence number: A's ID + unlocking time;

[0129] Door lock ID: extract A's ID;

[0130] Time: extract A's current clock;

[0131] Unlocker number: a's personnel number, extracted from A's check-in personnel registration list;

[0132] Unlocking person's face image / feature vector: a face image / feature vector;

[0133] Unlocking people's structured features: performing structured intelligent analysis and extraction based on a;

[0134] Companion 1 face image / feature vector: b face image / feature vector;

[0135] Confidence of peer 1's facial image / feature vector: graded according to three levels of confidence;

[0136] Companion 1 structured features: perform structured intelligent parsing and extraction based on b;

[0137] Face image / feature vector of companion 2: face image / feature vector of c;

[0138] Confidence of the facial image / feature vector of companion 2: graded according to three levels of confidence;

[0139] Companion 2 structured features: structured intelligent parsing and extraction based on c;

[0140] Face image / feature vector of companion 3: d face image / feature vector;

[0141] Confidence of the facial image / feature vector of the third companion: graded according to three levels of confidence;

[0142] Companion 3 structured features: perform structured intelligent parsing and extraction based on d;

[0143] Multicast message sending: According to the multicast group IP address uniformly configured by the platform, information about one-open-multiple-entry events is sent in the form of multicast. The payload of the multicast message includes the event sequence number, door lock ID, time, unlocker number, multiple peer face images / feature vectors, multiple peer face matching confidences, multiple peer structured features, etc.

[0144] For example, assume that smart lock A detects and identifies persons a, b, c, and d (where a is the person checking in, and b, c, and d are their companions). It then sends information such as the facial images / feature vectors, facial matching confidence, and structural features of the four persons. The facial matching confidence is also sent to other smart locks for reference.

[0145] (7) Multicast message sending and receiving: After the smart door lock completes face matching, it sends the paired multiple-entry event information of this lock to other door locks in the form of multicast. Other door locks receive and extract the multiple-entry event information in the multicast message and write it into the multiple-entry track list of this door lock, and compare the human image / structured features with the multiple-entry track list of this door lock. If the face matching confidence is higher than the track point in the multiple-entry track list, the face image / feature vector and face matching confidence of the track point with the largest human image / structured feature similarity value in the multiple-entry track list are updated.

[0146] Multicast Message Reception and Processing: After receiving a multicast message, the smart door lock in the multicast group extracts and stores the "multiple-entry" event information. It then compares the human images / structured features in the multicast message with those in the lock's "multiple-entry" trajectory list. If the face match confidence in the multicast message is higher, the corresponding face image / feature vector and face match confidence in the "multiple-entry" trajectory list with the highest similarity value greater than a threshold (typically set to 80%) are updated to the same value as the face image / feature vector and face match confidence in the multicast message. The "multiple-entry" event information extracted from the multicast message is written to the lock's "multiple-entry" trajectory list. The data includes the event sequence number, door lock ID, time, unlocker number, multiple accompanying facial images / feature vectors, multiple accompanying facial match confidences, and multiple accompanying structural features. The storage period is typically set to 10 minutes, with automatic deletion upon expiration.

[0147] Note: After calculating the similarity of human images / structured features, the similarity value must be greater than a threshold (generally set to 80%) and be the maximum. This is to update the person with the highest human image similarity in the one-open, multiple-entry trajectory list. Since the similarity threshold does not meet the requirement, it may not be the same person and should be excluded. Not all trajectory points that meet the threshold requirement are updated because as long as one trajectory point is updated, it is sufficient to match the face image / feature vector with the highest confidence when searching and matching multiple entries of the door lock. Multicast messages do not need to reply to the update status of this door lock because each door lock has synchronized all the spatiotemporal trajectory points of the multicast group members and can be updated separately. In principle, there is no situation where the face match confidence is higher than that in the newly received multicast message because each door lock has synchronized the one-open, multiple-entry trajectory list.

[0148] (8) Send a one-open-multiple-entry event message to the platform in unicast mode: According to the platform's requirements, all multi-entry persons are matched to the one-open-multiple-entry event of the face and sent to the platform.

[0149] There are two ways for the door lock to send the "one-time multiple-entry" event to the platform: one is quasi-real-time sending, in which the "one-time multiple-entry" event is sent immediately after real-time identification and real-time matching; the other is asynchronous sending, in which the "one-time multiple-entry" event is first cached (the delay length is set to 10 minutes), and the "one-time multiple-entry" event is updated based on possible cross-lock supplementary messages before being sent to the platform.

[0150] Near-real-time transmission: After A recognizes the facial images / feature vectors of all persons in a one-open, multiple-entry event, it unicasts a message to the platform. The message contains the following data: event sequence number, door lock ID, timestamp, unlocker ID, unlocker facial image / feature vector, unlocker structured features, facial images / feature vectors of multiple companions, facial match confidence levels for multiple companions, and structured features for multiple companions. The platform receives the message and stores and applies it according to its business processes.

[0151] Asynchronous sending: If A finds a data item with medium confidence after identifying all the face images / feature vectors of all persons in the one-open-multiple-entry event, it will send the corrected one-open-multiple-entry event information to the platform within the total delay period of 10 minutes according to the processing result of step (5).

[0152] Note: After a period of operation, the processing procedures of A, B and all other smart locks are the same. The procedures of A and B are used as examples above mainly to facilitate the description of the processing procedures of each procedure. There is a certain difference in real-time performance between quasi-real-time sending and asynchronous sending, but the recognition of multiple entries in asynchronous sending may be more accurate. After the platform receives the event message of multiple entries in the smart lock, it can carry out the control of mobile population, trajectory characterization, etc. The details of the application are not the focus of this invention.

[0153] It can be seen that within the capture time period, the facial information and body information of multiple people are extracted; if the facial image and body image are extracted from the same person image at the same time, the direct pairing is successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the one-open-multiple-entry trajectory list, the similarity of the body information is calculated, and if the similarity is greater than the preset threshold, the pairing is successful; for the body information that has not been successfully retrieved and paired, its spatiotemporal trajectory sequence is constructed, and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence; the various pairing results are fused, and for each face information corresponding to the human image information, the face matching with the highest confidence is taken as the fusion result, so that a confident identification of the results of multiple people can be constructed to realize the identification of multiple people.

[0154] Another embodiment of the present invention provides a face recognition system for a smart lock with multiple entries. Figure 3 , the system may include:

[0155] Monitoring module 301, for detecting multiple people: extracting facial information and body information of multiple people during the capture period;

[0156] Pairing module 302 is used for multi-mode pairing: direct pairing and exclusion pairing: if a face image and a body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if a unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by searching the one-open-multiple-entry trajectory list. If the similarity is greater than a preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not successfully retrieved and matched, its spatiotemporal trajectory sequence is constructed and compatibility is calculated with the face cluster spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence level;

[0157] The fusion module 303 is used for fusing the pairing results: fusing the pairing results, and taking the face matching result with the highest confidence level as the fusion result for each face information corresponding to the human image information.

[0158] It can be seen that within the capture time period, the facial information and body information of multiple people are extracted; if the facial image and body image are extracted from the same person image at the same time, the direct pairing is successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the one-open-multiple-entry trajectory list, the similarity of the body information is calculated, and if the similarity is greater than the preset threshold, the pairing is successful; for the body information that has not been successfully retrieved and paired, its spatiotemporal trajectory sequence is constructed, and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence; the various pairing results are fused, and for each face information corresponding to the human image information, the face matching with the highest confidence is taken as the fusion result, so that a confident identification of the results of multiple people can be constructed to realize the identification of multiple people.

[0159] An embodiment of the present invention further provides a storage medium storing a computer program, wherein the computer program is configured to execute the steps of any one of the above method embodiments when running.

[0160] Specifically, in this embodiment, the above-mentioned storage medium may be configured to store a computer program for performing the following steps:

[0161] S201, multiple-entry person detection: extracting facial information and body information of multiple-entry persons within the capture time period;

[0162] S202, multi-mode pairing: direct pairing and exclusion pairing: if a face image and a body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if a unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by searching the one-open-multiple-entry trajectory list. If the similarity is greater than a preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not successfully retrieved and paired, its spatiotemporal trajectory sequence is constructed and compatibility is calculated with the face cluster spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence level;

[0163] S203, pairing result fusion: fuse the pairing results, and for each face information corresponding to the human image information, take the face with the highest confidence level as the fusion result.

[0164] It can be seen that within the capture time period, the facial information and body information of multiple people are extracted; if the facial image and body image are extracted from the same person image at the same time, the direct pairing is successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the one-open-multiple-entry trajectory list, the similarity of the body information is calculated, and if the similarity is greater than the preset threshold, the pairing is successful; for the body information that has not been successfully retrieved and paired, its spatiotemporal trajectory sequence is constructed, and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence; the various pairing results are fused, and for each face information corresponding to the human image information, the face matching with the highest confidence is taken as the fusion result, so that a confident identification of the results of multiple people can be constructed to realize the identification of multiple people.

[0165] An embodiment of the present invention further provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the steps in any one of the above method embodiments.

[0166] Specifically, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0167] Specifically, in this embodiment, the processor may be configured to execute the following steps through a computer program:

[0168] S201, multiple-entry person detection: extracting facial information and body information of multiple-entry persons within the capture time period;

[0169] S202, multi-mode pairing: direct pairing and exclusion pairing: if a face image and a body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if a unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by searching the one-open-multiple-entry trajectory list. If the similarity is greater than a preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: for body information that is not successfully retrieved and paired, its spatiotemporal trajectory sequence is constructed and compatibility is calculated with the face cluster spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence level;

[0170] S203, pairing result fusion: fuse the pairing results, and for each face information corresponding to the human image information, take the face with the highest confidence level as the fusion result.

[0171] It can be seen that within the capture time period, the facial information and body information of multiple people are extracted; if the facial image and body image are extracted from the same person image at the same time, the direct pairing is successful; if the unique pairing relationship is determined by the elimination method, the pairing is successful; by retrieving the one-open-multiple-entry trajectory list, the similarity of the body information is calculated, and if the similarity is greater than the preset threshold, the pairing is successful; for the body information that has not been successfully retrieved and paired, its spatiotemporal trajectory sequence is constructed, and the compatibility calculation is performed with the face clustering spatiotemporal trajectory sequence in the one-open-multiple-entry trajectory list to determine the confidence; the various pairing results are fused, and for each face information corresponding to the human image information, the face matching with the highest confidence is taken as the fusion result, so that a confident identification of the results of multiple people can be constructed to realize the identification of multiple people.

[0172] The above describes in detail the structure, features and effects of the present invention based on the embodiments shown in the drawings. The above is only a preferred embodiment of the present invention, but the scope of implementation of the present invention is not limited to what is shown in the drawings. Any changes made in accordance with the concept of the present invention, or modifications to equivalent embodiments with equivalent changes, which do not exceed the spirit covered by the description and drawings, should be within the scope of protection of the present invention.

Claims

1. A face recognition method for a smart lock with multiple entries, characterized in that: The method comprises: Multiple-entry person detection: Extract facial and body information of multiple people within the capture period; Multi-mode pairing: direct pairing and exclusion pairing: if the face image and body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if the unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by searching the one-open-multiple-input trajectory list. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: based on the unmatched The human body image / structured features are used to construct its spatiotemporal trajectory sequence, including time trajectory sequence and spatial trajectory sequence. The constructed spatiotemporal trajectory sequence is compared with the face cluster spatiotemporal trajectory sequence in the one-open-multiple-input trajectory list to determine whether they completely overlap, partially overlap, or are completely compatible. If the spatiotemporal trajectories completely overlap, the confidence level is certain; if they partially overlap and are completely compatible, the confidence level is high; if they do not overlap but are completely compatible, the confidence level is medium. The successfully matched face image / feature vector and confidence level are updated to the current one-open-multiple-input event information. Fusion of pairing results: Fusion of each pairing result, taking the face image / feature vector with the highest confidence corresponding to each human image / structured feature; generating complete one-open-multiple-entry event information, including event serial number, door lock ID, time, unlocker number, multiple companion face images / feature vectors, multiple companion face matching confidences, and multiple companion body structured features; sending the one-open-multiple-entry event information to other smart locks in multicast form, and to the platform in unicast form at the same time; writing the one-open-multiple-entry event information into the one-open-multiple-entry track list of the smart lock, storing it for 10 minutes, and automatically deleting it after the expiration.

2. The method according to claim 1, characterized in that The direct pairing includes: Based on the video images captured by the smart lock, the face detection algorithm is used to extract the face image / feature vector, and the human body detection algorithm is used to extract the human body image / structural features; If both the facial image / feature vector and the body image / structural features are extracted from the same person image, the pairing is directly determined to be successful, and the confidence level is confirmed; If the direct matching method is not completely successful, the exclusion matching method is used to uniquely match the remaining facial images / feature vectors and body images / structured features. If the remaining facial images / feature vectors and body images / structured features are the only matching relationship, the matching is successful and the confidence level is certain.

3. The method according to claim 2, characterized in that The face retrieval pairing includes: For human images / structured features that are not directly matched successfully, the similarity between them and the human images / structured features in the list of one-open-multiple-entry trajectories stored in the smart lock is calculated; If the similarity is greater than the preset threshold, the corresponding face image / feature vector is extracted from the one-open-multiple-input trajectory list, and the pairing is successful with a medium or high confidence level; Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

4. A face recognition system for smart locks with multiple entries, characterized in that: The system comprises: Monitoring module, used for multiple-entry person detection: extracting facial and body information of multiple people within the capture period; Pairing module, used for multi-mode pairing: direct pairing and exclusion pairing: if the face image and body image are extracted from the same person image at the same time, the direct pairing is successful and the confidence level is certain; if the unique pairing relationship is determined by the elimination method, the pairing is successful and the confidence level is certain; face retrieval pairing: for body information that is not directly matched or excluded, the similarity of the body information is calculated by searching the one-open-multiple-input trajectory list. If the similarity is greater than the preset threshold, the pairing is successful and the confidence level is medium or high; spatiotemporal trajectory compatibility pairing: based on the unpaired For successful human images / structured features, construct their spatiotemporal trajectory sequences, including time trajectory sequences and spatial trajectory sequences; compare the constructed spatiotemporal trajectory sequences with the face clustering spatiotemporal trajectory sequences in the one-open-multiple-input trajectory list to determine whether they completely overlap, partially overlap, or are completely compatible; if the spatiotemporal trajectories completely overlap, the confidence level is certain; if they partially overlap and are completely compatible, the confidence level is high; if they do not overlap but are completely compatible, the confidence level is medium; update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information; The fusion module is used to fuse the pairing results: fuse the various pairing results, and take the facial image / feature vector with the highest confidence corresponding to each human image / structured feature; generate complete one-open-multiple-entry event information, which includes the event sequence number, door lock ID, time, unlocker number, multiple companion face images / feature vectors, multiple companion face matching confidences, and multiple companion body structured features; send the one-open-multiple-entry event information to other smart locks in multicast form and to the platform in unicast form at the same time; write the one-open-multiple-entry event information into the one-open-multiple-entry track list of the smart lock, store it for 10 minutes, and automatically delete it after the expiration.

5. The system according to claim 4, characterized in that The pairing module is specifically used to: Based on the video images captured by the smart lock, the face detection algorithm is used to extract the face image / feature vector, and the human body detection algorithm is used to extract the human body image / structural features; If both the facial image / feature vector and the body image / structural features are extracted from the same person image, the pairing is directly determined to be successful, and the confidence level is confirmed; If the direct matching method is not completely successful, the exclusion matching method is used to uniquely match the remaining facial images / feature vectors and body images / structured features. If the remaining facial images / feature vectors and body images / structured features are the only matching relationship, the matching is successful and the confidence level is certain.

6. The system according to claim 5, characterized in that The pairing module is specifically used to: For human images / structured features that are not directly matched successfully, the similarity between them and the human images / structured features in the list of one-open-multiple-entry trajectories stored in the smart lock is calculated; If the similarity is greater than the preset threshold, the corresponding face image / feature vector is extracted from the one-open-multiple-input trajectory list, and the pairing is successful with a medium or high confidence level; Update the successfully matched face image / feature vector and confidence level to the current one-open-multiple-input event information.

7. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the method according to any one of claims 1 to 3 when run.

8. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 3.

Images