A method and system for face privacy processing of an intelligent lock

Abstract

The present invention discloses a method and system for face privacy processing of an intelligent lock. The method includes: Intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other intelligent locks receive the multicast message, they compare the unrecognized face images, and if the comparison is successful, face privacy processing is performed; Intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, a query request is sent to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; Intelligent lock A integrates the reply messages and privacy processing results, encodes and generates a video image data stream, and sends the video image data stream to the platform for decoding processing. Using the embodiments of the present invention, the protection of the face privacy of the property owner can be ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent locks, and particularly relates to a method and system for face privacy processing of intelligent locks. Background Art

[0002] With the needs of community cluster management and social governance, it has become increasingly common for householders to install intelligent door locks that can be remotely networked. The intelligent door locks are integrated into the remote management platform through the Internet. Householders can unlock and open the door through the face recognition technology of the intelligent door locks, which greatly facilitates the householders. The management agency can remotely monitor the video images of the people at the door through the remote management platform.

[0003] However, there is a risk of leakage of the face privacy of the householder in the face images transmitted to the remote management platform through the Internet. In order to protect the personal privacy in the face images, active protection measures should be taken for the privacy of the face video images. In actual management work, the management agency does not pay attention to the face images of the householder that can be normally recognized and compared in the intelligent lock (only needs to know who opened the door for management needs), and what it really cares about is the access situation of non-household members (such as strangers, visitors, etc., who may pose security risks). Summary of the Invention

[0004] The purpose of the present invention is to provide a method and system for face privacy processing of intelligent locks to solve the deficiencies in the prior art and ensure the protection of the face privacy of the householder.

[0005] An embodiment of the present application provides a method for face privacy processing of intelligent locks, and the method includes:

[0006] Face detection and privacy processing: Intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the uncompared face images are sent to other intelligent locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other intelligent locks compare the uncompared face images with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in the form of unicast;

[0007] Message check and reminder: Intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; Image integration and encoding: Intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock, and encodes to generate a video image data stream containing descriptive metadata; Intelligent lock A sends the integrated video image data stream to the platform for decoding processing.

[0008] Optionally, the face detection and privacy processing includes:

[0009] The intelligent lock A detects multiple faces through video images, performs face segmentation, and labels the face image ID and position coordinates.

[0010] Calculate the similarity between the segmented face image and the list of family members of this lock. If the similarity exceeds the threshold, replace the face image with the person ID.

[0011] If the comparison fails, send the uncompared face image and face image ID to other intelligent locks in the community in the form of multicast.

[0012] Optionally, the collaborative face privacy processing includes:

[0013] The intelligent lock B receives the multicast message, extracts the uncompared face image, and compares it with the list of family members of this lock.

[0014] If the comparison is successful, replace the face image with the person ID and reply the processing result in the form of unicast.

[0015] If the comparison fails, reply the uncompared information in the form of unicast.

[0016] Optionally, the message check and reminder include:

[0017] The intelligent lock A counts the multicast reply messages and checks whether there are intelligent locks that have not replied.

[0018] If there are intelligent locks that have not replied, send a query request to the independent relay gateway.

[0019] The independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check.

[0020] Optionally, the image integration and encoding include:

[0021] The intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock.

[0022] Replace the face image with the person ID and perform blanking processing on the panoramic image.

[0023] Generate a video image data stream containing descriptive metadata and embed the metadata into the encoded data.

[0024] Another embodiment of the present application provides a system for intelligent lock face privacy processing. The system includes:

[0025] Detection module, for face detection and privacy processing: Smart Lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the unmatched face images are sent to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other smart locks compare the unmatched face images with the list of family members of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in unicast form;

[0026] Processing module, for message checking and urging: Smart Lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the smart locks that have passed the health check; Image integration and encoding: Smart Lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Smart Lock A sends the integrated video image data stream to the platform for decoding processing.

[0027] Another embodiment of the present application provides a storage medium, in which a computer program is stored. Among them, the computer program is set to execute the method described in any one of the above when running.

[0028] Another embodiment of the present application provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is set to run the computer program to execute the method described in any one of the above.

[0029] Compared with the prior art, a method for face privacy processing of a smart lock provided by the present invention. Smart Lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after receiving the multicast message, other smart locks compare the unmatched face images. If the comparison is successful, face privacy processing is performed; Smart Lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the smart locks that have passed the health check; Smart Lock A integrates the reply messages and privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby being able to ensure the protection of the face privacy of the property owner. Description of the Drawings

[0030] Figure 1 It is a hardware structure block diagram of a computer terminal for a method of face privacy processing of a smart lock provided by an embodiment of the present invention;

[0031] Figure 2Schematic flowchart of a method for face privacy processing of an intelligent lock provided by an embodiment of the present invention;

[0032] Figure 3 Schematic structural diagram of a system for face privacy processing of an intelligent lock provided by an embodiment of the present invention. Detailed implementation manners

[0033] The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain the present invention, and should not be construed as limiting the present invention.

[0034] An embodiment of the present invention first provides a method for face privacy processing of an intelligent lock. This method can be applied to an electronic device, such as a computer terminal, specifically, a general computer, etc.

[0035] The following takes running on a computer terminal as an example to describe it in detail. Figure 1 Hardware structure block diagram of a computer terminal for a method for face privacy processing of an intelligent lock provided by an embodiment of the present invention. As Figure 1 shown, this computer device includes a processor, a memory, and a network interface connected through a system bus. Among them, the memory can include a non-volatile storage medium and an internal memory.

[0036] The non-volatile storage medium can store an operating system and a computer program. This computer program includes program instructions. When the program instructions are executed, the processor can execute any method for face privacy processing of an intelligent lock.

[0037] The processor is used to provide computing and control capabilities to support the operation of the entire computer device.

[0038] The internal memory provides an environment for the operation of the computer program in the non-volatile storage medium. When this computer program is executed by the processor, the processor can execute any method for face privacy processing of an intelligent lock.

[0039] This network interface is used for network communication, such as sending assigned tasks, etc. Those skilled in the art can understand that Figure 1 the structure shown in

[0040] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0041] See Figure 2 , an embodiment of the present invention provides a method for intelligent lock face privacy processing, which may include the following steps:

[0042] S201, face detection and privacy processing: Intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the uncompared face image is sent to other intelligent locks in the form of a multicast message for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other intelligent locks compare the uncompared face image with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in unicast form; Specifically, the face detection and privacy processing includes:

[0043] Intelligent lock A detects multiple faces through video images, performs face segmentation and marks the face image ID and position coordinates; calculates the similarity between the segmented face image and the family member list of this lock. If the similarity exceeds the threshold, the face image is replaced with the person ID; if the comparison fails, the uncompared face image and the face image ID are sent to other intelligent locks in the community in multicast form.

[0044] Specifically, the collaborative face privacy processing includes: Intelligent lock B receives the multicast message, extracts the uncompared face image and compares it with the family member list of this lock; if the comparison is successful, the face image is replaced with the person ID, and the processing result is replied in unicast form; if the comparison fails, the uncompared information is replied in unicast form.

[0045] S202, Message Check and Urging: Smart Lock A counts the multicast reply messages. If there are un-responded smart locks, it sends a query request to the independent relay gateway; the independent relay gateway conducts a health check and sends an urging reply message to the smart locks that pass the health check; Image Integration and Encoding: Smart Lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Smart Lock A sends the integrated video image data stream to the platform for decoding processing. Specifically, the message check and urging include:

[0046] Smart Lock A counts the multicast reply messages and checks if there are un-responded smart locks; if there are un-responded smart locks, it sends a query request to the independent relay gateway; the independent relay gateway conducts a health check and sends an urging reply message to the smart locks that pass the health check.

[0047] Specifically, the image integration and encoding include: Smart Lock A integrates the reply messages from other smart locks and the privacy processing results of this lock; replaces the face image with the personnel ID and leaves the panoramic image blank; generates a video image data stream containing descriptive metadata and embeds the metadata into the encoded data.

[0048] Smart Lock: It refers to a lock improved on the basis of a traditional mechanical lock, which is more intelligent and simpler in terms of user security, identification, and management. The smart door lock mentioned in this article has the function of face recognition for opening the door.

[0049] Privacy Computing: It is a technology that realizes data analysis and calculation on the premise of ensuring that data is not leaked externally. The privacy computing mentioned in this article is used for privacy protection of face images collected by smart door locks during Internet transmission and remote management.

[0050] Lock Event: It refers to an event detected and recognized by a smart lock. The smart lock extracts the information related to this event as lock event information, which includes lock event serial number, door lock ID, time, personnel identity information, face pictures, video clips, etc.

[0051] The purpose of the present invention is to propose a method for face privacy protection for community owners. By multicasting to other smart locks in the community to perform face privacy processing for their own household members respectively, and at the same time adopting the processing of multicast reply messages and the urging reply of the independent relay gateway to ensure that all smart locks complete the face privacy processing for their own household members, thereby realizing the face privacy protection of all community owners.

[0052] The present invention proposes a method. After the face image in the intelligent lock is replaced by face removal in this lock, it is sent to other intelligent locks for face removal replacement in the form of multicast. When not all multicast messages are replied, a reminder message is sent after querying the IP address of the independent relay gateway, and the health check of the intelligent lock is performed and the results are updated in real time to all intelligent locks. The complete process of a technical implementation is as follows:

[0053] (1)Preconditions:

[0054] Intelligent lock: An intelligent lock is installed on the household entrance door of the resident (each lock has an independent intelligent lock ID, intelligent lock IP address, family member list, and community door lock list), and is connected to the remote management platform (for convenience of description, hereinafter referred to as the platform), registered with the platform and updated periodically.

[0055] Explanation: The family member list is the list of household members stored in the intelligent lock. The scope of this list only includes the registered residents of this door lock, including unique person IDs, names, face images collected during registration, etc. Among them, the person ID is a unique number generated by the platform for each person in the community. The person ID must be resolved in the platform to query the real identity information. The family member list is updated and issued by the platform regularly; the community door lock list is a collection of intelligent door locks constructed and updated regularly by the independent relay gateway, including multicast IP addresses, intelligent lock ID numbers, etc.

[0056] Platform: Maintains a list of community personnel (data items include door lock ID, door lock geographical address, person ID, name, face image collected during registration, etc. of family members), and saves the corresponding relationship between the face images and person IDs of each formal owner in the community.

[0057] Explanation: In order to better manage the floating tenants in the community, the door lock needs to send the picture to the platform every time a face is swiped to open the door, and the platform records the entry and exit situations of each tenant. However, in order to prevent ordinary management personnel from illegally learning the entry and exit privacy of formal owners, the pictures reported by the door lock to the platform must be privacy-processed, and the faces of formal owners are hidden.

[0058] Independent relay gateway: Saves the community door lock list (data items include door lock ID, door lock IP address, etc.), saves the corresponding relationship between the door lock ID and the door lock IP, and the independent relay gateway can also receive multicast messages.

[0059] (2)Face detection and face privacy processing of intelligent lock A: After the door lock A detects multiple faces in the video image, it performs face segmentation, recognition and comparison. If the comparison and recognition are for family members, it unlocks and performs face privacy processing for this family member. If a non-family member is recognized, it is sent to other intelligent locks in the form of multicast for collaborative face privacy processing.

[0060] Description: For the sake of convenience in description, the intelligent lock that initiates multicast will be abbreviated as intelligent lock A hereinafter, the intelligent lock of the community door lock will be abbreviated as intelligent lock B, the intelligent locks that do not reply to the multicast message in time will be abbreviated as intelligent locks E, F, and G, and the intelligent locks with failed health detection will be abbreviated as intelligent locks E and F.

[0061] Face detection and segmentation: When multiple people appear in front of the intelligent lock, after the intelligent lock detects a face through the intelligent detection and recognition algorithm, it performs face segmentation to form multiple face images and performs annotation. The labels include the lock event ID (which refers to the event of collecting a face, and this serial number can adopt the door lock ID + time + serial number), door lock ID, time, panoramic image ID (the serial number extracted from the video frame, and the number can adopt the lock event serial number + serial number), panoramic image, face image ID (the ID number is unique and is the serial number assigned when segmenting the face area image from the panoramic image, and the number can adopt the lock event serial number + panoramic image ID + serial number), face image position coordinates (upper left X, Y coordinates, lower right X, Y coordinates), face image);

[0062] Description: In order to distinguish the segmented multiple face images and perform image re-encoding, it is necessary to locate and annotate them; if only 1 person is detected and recognized in the video image, it is processed according to the special case of multiple people; assume that the people detected in the current image are four people, namely a, b, c, and d.

[0063] Face privacy processing: Calculate the similarity between the segmented face image and the face images in the door lock family member list. When the similarity meets the threshold (generally set to 80%), it indicates a successful comparison. Extract the person ID number of the face image in the comparison from the family member list to form comparison information including: lock event ID, panoramic image ID, whether it is compared, compared face image ID, compared face image coordinates, compared face image, compared person ID.

[0064] Description: When this lock performs a face comparison and it is successful, it means that this person is a family member of this lock. In order to protect the face privacy of the formal owners (family members), the person ID number is used as a label instead. Other door locks cannot resolve the true identity of the person by obtaining this person ID. Only the platform can obtain the true identity information of the person with this person ID according to the corresponding relationship of the community personnel list under authorization; the uncompared face image is not a family member of this lock and may be a family member of other door locks, and it is necessary to perform face privacy protection processing for the formal owners (family members) of other door locks; assume that the person recognized by intelligent lock A is a.

[0065] Multicast sending collaborative face privacy processing request message: After the intelligent lock A performs face privacy processing, if there is a face image that fails to match, the unmatched face image and the face image ID are sent to other intelligent locks in the community in the form of multicast for collaborative face privacy processing (for the face privacy processing of the family members of the door lock). The message content of the multicast message includes the intelligent lock ID, the lock event ID, the face image ID, and the face image.

[0066] Note: Multicast is a network communication method that allows one or more senders / community door locks (intelligent door locks) to send the same data packet to multiple receivers (intelligent door locks). That is, the sender only needs to send once, and other members can receive it. Multicast uses a specific multicast address to identify a group of receivers. The present invention does not care about the multicast technology, but only uses the multicast technology and applies multicast through the API of the network management system (NMS) or a dedicated multicast management software.

[0067] (3)Intelligent lock B performs collaborative face privacy processing: Intelligent lock B receives the multicast message of the collaborative face privacy processing request sent by A, extracts the face image in the multicast message packet and compares it with the face images in the family member list of B. If it matches, the face image is replaced with the person ID number, and the processing result of the collaborative face privacy processing request is replied in the form of unicast.

[0068] Receiving the multicast message: Intelligent lock B receives the multicast message sent by intelligent lock A, extracts the lock event ID, the face image ID, the face image from the multicast message packet, and records the door lock IP address of intelligent lock A.

[0069] Note: There may be multiple face image IDs and their corresponding face images received by intelligent lock B from the multicast message packet, which need to be extracted according to the corresponding relationship. Otherwise, it will cause the wrong splicing position when intelligent lock A splices the images.

[0070] Face privacy processing: Intelligent lock B calculates the similarity between the received face image and the face images in the family member list of door lock B. When the similarity meets the threshold (usually set to 80%), it means the comparison is successful. The person ID number corresponding to the face image is extracted from the family member list, and the comparison information includes: the lock event ID, whether it matches, the matching face image ID, and the matching person ID.

[0071] Note: If the comparison is successful, it means that the face image is a family member of intelligent lock B, and face privacy processing should be performed (replaced with the person ID). To prevent intelligent lock A from knowing the person's identity information, after replacing it with the person ID number, A still cannot resolve the person's identity information because it does not have a person ID and person identity comparison table. Assume the person recognized by intelligent lock B is b.

[0072] Unicast reply for face privacy processing result: There are two cases for the comparison result of Smart Lock B, which need to be processed separately. The first case is no match. The comparison information is replied to the multicast message in unicast form (both the source IP address and the source MAC address are randomly filled), including the ID of Smart Lock B, the lock event ID of the received multicast message, and whether there is a match / no. The second case is a match. The comparison information is replied to the multicast message in unicast form (both the source IP address and the source MAC address are randomly filled), including the ID of Smart Lock B, the lock event ID in the received multicast message, whether there is a match / yes, the face image ID of the match, and the person ID of the match (if there are multiple people, multiple pairs of face image IDs of the match and person IDs of the match must be replied).

[0073] Note: Replying to the collaborative face privacy processing request in unicast form is to reduce the flooding of multicast messages in the network; randomly filling both the source IP address and the source MAC address is to prevent Smart Lock A from guessing the door lock information of B through the source IP address and the source MAC address; if multiple face comparisons are successful, the face image ID of the match and the person ID of the match must be paired. For example, face image ID (1) corresponds to person ID (a), and face image ID (2) corresponds to person ID (b). Otherwise, Smart Lock A cannot accurately pair after receiving the reply message.

[0074] (4) Processing of the reply to the collaborative face privacy processing request by Smart Lock A: Smart Lock A receives the unicast reply message from Smart Lock B and counts whether there is any missing multicast reply message. If there is a missing message, a query request is sent to the independent relay gateway.

[0075] Receiving the multicast reply message for the collaborative face privacy processing request: After Smart Lock A receives the reply message from B for the collaborative face privacy processing request sent by A, it extracts the comparison information in the message packet, including the ID of Smart Lock B, the lock event ID, whether there is a match, the face image ID of the match, and the person ID of the match, and then processes it. First, perform a reply message statistics and record the lock event ID / door lock ID / reply message sequence number (numbered in the order of reception); then process the comparison information. If whether there is a match is "no", do not process it. If whether there is a match is "yes", then extract the comparison message of Smart Lock B (the ID of Smart Lock B, the lock event ID, whether there is a match, the face image ID of the match, and the person ID of the match).

[0076] Description: If the reply message of B is received, the reply message statistics must be performed according to the lock event ID. Since the multicast messages sent with different lock event IDs may have different reply situations, the reply message statistics is to check whether all door locks have replied; the method of reply message statistics is to sequentially number them in the order of receiving the reply messages, that is, the reply message sequence number is incremented by 1 when each reply message is received; if the match is "yes", the face image ID and personnel ID need to be extracted according to the corresponding relationship.

[0077] Check for missing multicast reply messages: Smart lock A starts timing after sending the multicast message for collaborative face privacy processing request. After a delay of 5 seconds, it checks the reply message statistics situation, compares the recorded lock event ID / door lock ID / reply message sequence number with the door lock IDs in the community door lock list one by one to check whether all have replied. If all have replied, it proceeds to step (7). If there are unreplied messages, first check if there are invalid smart lock IDs (sent by the independent relay gateway) and exclude them, then extract the unreplied door lock IDs from the community door lock list as the set of unreplied door lock IDs (assuming the unreplied door locks are E, F, G), and the data items include the lock event ID and the unreplied door lock ID.

[0078] Description: After the multicast message is sent, due to network connection failures, device offline, etc. of the smart door locks, smart lock A may not receive all the expected replies (all smart locks in the community door lock list).

[0079] Unicast a smart lock query request for unreplied messages to the independent relay gateway: Smart lock A unicasts a smart lock query request for unreplied messages to the independent relay gateway, including the ID of smart lock A, the lock event ID, the unreplied door lock IDs (E, F, G), etc.

[0080] Description: The unicast smart lock query request message for unreplied messages to the independent relay gateway is to send a further query for the smart locks with unreplied messages, and the independent relay gateway is to notify each smart lock that has not replied to the multicast message to reply again.

[0081] (5)Processing of the intelligent lock query request message for un-responded messages sent by A by the independent transit gateway: The independent transit gateway receives the intelligent lock query request message for un-responded messages sent by intelligent lock A, queries the corresponding door lock IP address in its community member list (data items include door lock ID, door lock IP address, etc.) according to the door lock ID of the un-responded message, and performs an intelligent lock health check (such as in the way of ping). If the health check is successful (such as ping is successful), unicast urging response messages are sent one by one to notify the un-responded intelligent lock to reply. If the health check fails (such as ping fails), the intelligent lock ID with health check failure is synchronized to all intelligent locks including the community door lock list or replied to intelligent lock A.

[0082] Receiving and checking the intelligent lock with un-responded messages: The independent transit gateway receives the intelligent lock query request message for the un-recovered messages of intelligent lock A, and extracts relevant information, including the ID of intelligent lock A, lock event ID, un-responded intelligent lock IDs (E, F, G), etc.

[0083] Note: When the independent transit gateway extracts relevant information from the intelligent lock query request message for un-responded messages of intelligent lock A, it needs to extract the ID of intelligent lock A and the lock event ID at the same time, because the independent transit gateway may receive query requests from many other intelligent locks.

[0084] Performing a health check on the intelligent lock: The independent transit gateway performs a health check (generally in the way of ping) on intelligent locks (E, F, G). If the health checks of intelligent locks E and F fail (such as intelligent locks E and F cannot be pinged, and the door locks that cannot be pinged are simply called failed door locks), notify intelligent lock A, and adopt two information sharing schemes. If the health check of intelligent lock G is successful (such as ping is successful), then send a urging response message to G.

[0085] Note: The health check generally adopts the method of executing when there is a request, and does not adopt the periodic method, because the periodic check will periodically wake up all intelligent locks, causing additional power consumption of the door locks.

[0086] Intelligent lock E and F failure information sharing scheme one: The independent transit gateway caches the information of the failure of intelligent locks E and F, and the validity period is 5 minutes, and it is deleted after the validity period expires; if at this time door lock Z detects the presence of multiple people and multicasts a collaborative face privacy processing request, and finds that door locks E and F do not give feedback and sends it to the transit gateway, the transit gateway judges the local cached failed door lock information. If it finds that there are failed door locks E and F, it directly replies to door lock Z that "E and F are failed". If it finds that there is no longer any failed door lock information (at this time it may be deleted after the validity period expires), then performs a health check on E and F. If it is still found that the check fails, then feedback to door lock Z that "E and F are failed".

[0087] Intelligent Lock E and F Failure Information Sharing Solution 2: The relay gateway broadcasts the information of failed door locks E and F to all door locks, specifying a validity period of 5 minutes. All intelligent locks that receive the information of the failed door locks only cache it for 5 minutes and delete it after the validity period expires; if at this time Door Lock Z detects the presence of multiple people and multicasts a collaborative face privacy processing request, and finds that Door Locks E and F do not respond, it checks the information of the failed door locks cached locally in Z. If there is still information about failed door locks E and F, it does not need to send it to the relay gateway again; if there is no information about failed door locks E and F, it sends it to the relay gateway, and the relay gateway performs a health check on E and F. If it is found that they are still failed, it feedbacks to Door Lock Z that "E and F are failed".

[0088] (6) Intelligent Lock G Receives and Processes the Urging Reply Message from the Independent Relay Gateway: Intelligent Lock G receives the urging reply message unicast by the independent relay gateway, extracts the face image in the urging reply message and compares it with the face images in the family member list of this lock. If there is a match, it replaces the face image with the person ID number and unicasts the processing result of the collaborative face privacy processing request to A.

[0089] Receiving Unicast Message: Intelligent Lock G receives the urging reply message from the independent relay gateway, extracts the lock event ID, face image ID, face image, and the IP address of Intelligent Lock A.

[0090] Explanation: Intelligent Lock G may have already processed the collaborative face privacy processing request of Intelligent Lock A, but the message may be lost during the process of sending the reply message. When receiving the urging reply message sent by the independent relay gateway, it processes it again as a new message; the door lock IP address of Intelligent Lock A needs to be extracted from the message, rather than from the source IP address of the message sent by the independent relay gateway (because the source IP address extracted here is the IP address of the independent relay gateway).

[0091] Face Privacy Processing: Process according to the method in step (3). Assume that the person identified by Intelligent Lock G is c.

[0092] Unicast Reply of Face Privacy Processing Result: Process according to the method in step (3).

[0093] Explanation: At this time, the reply message should be replied according to the IP address of Intelligent Lock A extracted from the urging reply message of the independent relay gateway.

[0094] (7) The intelligent lock A integrates the reply information of the face privacy processing of each intelligent lock: The intelligent lock A receives the unicast reply message from the intelligent lock B, the reply message of the intelligent lock G according to the urging request of the independent relay gateway, the set of intelligent locks with failed health check of the independent relay gateway, and the face privacy processing result completed by the intelligent lock A for integration. Respectively, replace the face image corresponding to the face image ID with the person ID after face privacy processing, and then re-encode and compress it into a video image data stream.

[0095] Reply message of intelligent lock B: The reply message of intelligent lock B is the reply to the multicast sent collaborative face privacy processing request by intelligent lock A. Extract the information that has undergone face privacy processing, including the ID of intelligent lock B, the lock event ID in the received multicast message, whether it is matched / yes, the face image ID, and the person ID.

[0096] Note: The situation where intelligent lock B is not matched during face privacy processing has been processed in step (4). Here, receive the information that has been matched and the person ID is given; assume that the person matched by intelligent lock B is b, and extract the lock event ID, face image ID (b), and person ID (b).

[0097] Reply message of intelligent lock G: The reply message of intelligent lock G is the reply from the independent relay gateway to intelligent lock A in the form of a unicast urging request according to the collaborative face privacy processing request of intelligent lock A. Extract the information that has undergone face privacy processing, including the ID of intelligent lock G, the lock event ID in the received multicast message, whether it is matched / yes, the face image ID, and the person ID.

[0098] Note: The situation where intelligent lock G is not matched during face privacy processing has been processed in step (6). Here, receive the information that has been matched and the person ID is given; assume that the person matched by intelligent lock G is c, and extract the lock event ID, face image ID (c), and person ID (c).

[0099] Set of time-limited door locks sent by the independent relay gateway: The intelligent lock A receives the information of the failed intelligent locks with failed health check from the independent relay gateway (including the door lock ID numbers with failed health check. Assume that the door locks are E and F at this time).

[0100] Comparison information of face privacy processing of intelligent lock A: According to step (2), the person identified by intelligent lock A through face privacy processing is person a. Extract the lock event ID of this person, face image ID (a), person ID (a), face image coordinates (a), and the panoramic image. (The comparison information of intelligent lock A includes: lock event ID, panoramic image ID, whether it is matched, the matched face image ID, the matched face image coordinates, the matched face image, and the matched person ID).

[0101] Check the completeness of the reply messages: The intelligent lock A checks one by one with the list of community door locks whether it is complete according to the direct reply to the multicast message received, the reply after urging through the independent relay gateway, and the intelligent lock notified by the independent relay gateway that the health check fails. If it is not complete (if the intelligent lock D is missing) and the message of the intelligent lock D is still not received after waiting for 2 seconds (it is possible that the independent relay gateway sent a reminder reply through the health check, but there may still be a situation where the reply message is lost), then ignore the intelligent lock D and enter the image integration processing process.

[0102] Image integration processing method: Segment and leave blank the panoramic image of the homeowner's face area, perform encoding compression, and generate descriptive metadata of the position coordinates of the face image segmentation area and the personnel ID and embed it into the encoded data to form the final video image data stream.

[0103] 1) According to the face image ID and the corresponding face image position coordinates (upper left corner X, Y coordinates, lower right corner X, Y coordinates) segmented from the panoramic image collected in step (2), the intelligent lock A performs a blanking process on the corresponding area in the panoramic image (usually filled with a specific color or transparent color, or set to an invalid pixel value);

[0104] 2) Generate descriptive metadata from the face image ID, the corresponding face image position coordinates (upper left corner X, Y coordinates, lower right corner X, Y coordinates), and the personnel ID given after face privacy processing;

[0105] 3) Re-encode the blanked image (usually using a video image encoder that supports metadata, such as FFmpeg, etc.), and embed the descriptive metadata into the video image data stream (usually the API of the FFmpeg encoder can be used).

[0106] Example: Suppose there are 4 faces in the image X of the lock event ID collected by the intelligent lock A in step (2), and the segmented face image IDs are face (1), face (2), face (3), and face (4). After the intelligent lock A recognizes that face (1) is person (a), the intelligent lock B recognizes that face (2) is person (b), and the intelligent lock G recognizes that face (3) is person (c), during integration, person (a), person (b), person (c) and their corresponding face image position coordinates are used as descriptive metadata (while face (4) is not recognized as the homeowner by any intelligent lock, so this face is retained) and embedded into the encoded data.

[0107] (8) The intelligent lock A sends the pictures processed for face privacy to the platform: It sends the image of the lock event ID integrated by the intelligent lock A to the platform. After decoding by the platform, the corresponding personnel ID is displayed in each face segmentation area. The authorized personnel can query the list of community personnel (data items include the door lock ID, the geographical address of the door lock, the personnel ID, name, and face image collected during registration of family members, etc.) according to the personnel ID to obtain their identity information.

[0108] The intelligent lock A sends the image of the lock event ID: The intelligent lock sends the image of the lock event ID (which has been encoded into a video image data stream) to the platform in unicast form.

[0109] The platform receives and decodes: After the platform receives the image data stream of the intelligent lock A, through video image decoding (using a video image decoder that supports metadata), it extracts descriptive metadata, identifies the blank area according to the face image position coordinates in the metadata, and uses image synthesis technology (using the FFmpeg video encoder that supports text overlay through API) to add the corresponding personnel ID to this blank area. For the faces of non-owner personnel in the video image, there is no segmentation and blanking process, and the faces can be normally displayed after video image decoding.

[0110] Note: The descriptive metadata embedded during encoding contains the position coordinates of the segmented face image and the personnel ID number after face privacy processing. After video image decoding, the corresponding area can be located and the personnel ID number can be overlaid based on this.

[0111] The platform authorizes user personnel identity query: According to the personnel ID displayed in the video image, querying the list of community personnel (data items include the door lock ID, the geographical address of the door lock, the personnel ID, name, and face image collected during registration of family members, etc.) can obtain their identity information, such as the name, door lock geographical address, face image collected during registration, etc. corresponding to the personnel ID.

[0112] Note: At all links of the entire process received from the intelligent lock, the independent relay gateway, and the platform, the identity information of the personnel cannot be known. Only the platform stores the corresponding relationship between the personnel ID and the identity information (the list of community personnel). To protect the face privacy of the property owners, unauthorized platform users should not query the personnel identity information.

[0113] The intelligent lock uses multicast to cooperate with neighboring intelligent locks, and removes the face images of the residents and neighbors in the video images captured and sent to the platform, realizing the protection of the personal privacy of the residents and neighbors. Since the independent relay gateway is responsible for processing, ordinary staff, the independent relay gateway, and the door lock A on the platform without sufficient authorization cannot see which face belongs to which household, ensuring the protection of the face privacy of the property owners.

[0114] It can be seen that the intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other intelligent locks receive the multicast message, they compare the uncompared face images, and if the comparison is successful, face privacy processing is performed; the intelligent lock A counts the multicast reply messages, and if there are intelligent locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; the intelligent lock A integrates the reply messages and the privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby ensuring the protection of the face privacy of the property owner.

[0115] Another embodiment of the present invention provides a system for intelligent lock face privacy processing, as shown in Figure 3 , the system may include:

[0116] A detection module 301 for face detection and privacy processing: The intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, it sends the uncompared face images to other intelligent locks in the form of a multicast message for collaborative privacy processing; Collaborative face privacy processing: After other intelligent locks receive the multicast message, they compare the uncompared face images with the list of family members of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in unicast form.

[0117] A processing module 302 for message checking and reminder: The intelligent lock A counts the multicast reply messages. If there are intelligent locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; Image integration and encoding: The intelligent lock A integrates the reply messages from other intelligent locks and the privacy processing results of this lock, encodes them to generate a video image data stream containing descriptive metadata; The intelligent lock A sends the integrated video image data stream to the platform for decoding processing.

[0118] It can be seen that the intelligent lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other intelligent locks receive the multicast message, they compare the uncompared face images, and if the comparison is successful, face privacy processing is performed; the intelligent lock A counts the multicast reply messages, and if there are intelligent locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the intelligent locks that have passed the health check; the intelligent lock A integrates the reply messages and the privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby ensuring the protection of the face privacy of the property owner.

[0119] An embodiment of the present invention further provides a storage medium, in which a computer program is stored. Wherein, the computer program is configured to execute the steps in any one of the above method embodiments when running.

[0120] Specifically, in this embodiment, the above storage medium may be configured to store a computer program for executing the following steps:

[0121] S201, Face detection and privacy processing: Smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the unmatched face images are sent to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other smart locks compare the unmatched face images with the family member list of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in unicast form;

[0122] S202, Message check and reminder: Smart lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Image integration and encoding: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Smart lock A sends the integrated video image data stream to the platform for decoding processing.

[0123] It can be seen that smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after other smart locks receive the multicast message, they compare the unmatched face images. If the comparison is successful, face privacy processing is performed; Smart lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Smart lock A integrates the reply messages and privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby being able to ensure the protection of the face privacy of the property owner.

[0124] An embodiment of the present invention further provides an electronic device, including a memory and a processor. A computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0125] Specifically, the above electronic device may further include a transmission device and an input / output device. Among them, the transmission device is connected to the above processor, and the input / output device is connected to the above processor.

[0126] Specifically, in this embodiment, the above-mentioned processor can be set to execute the following steps through a computer program:

[0127] S201, Face Detection and Privacy Processing: Smart Lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the uncompared face images are sent to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative Face Privacy Processing: After receiving the multicast message, other smart locks compare the uncompared face images with the list of family members of this lock. If the comparison is successful, face privacy processing is performed, and the processing result is replied in unicast form;

[0128] S202, Message Check and Urging: Smart Lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Image Integration and Encoding: Smart Lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Smart Lock A sends the integrated video image data stream to the platform for decoding processing.

[0129] It can be seen that Smart Lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, collaborative privacy processing is performed; after receiving the multicast message, other smart locks compare the uncompared face images. If the comparison is successful, face privacy processing is performed; Smart Lock A counts the multicast reply messages. If there are smart locks that have not replied, a query request is sent to the independent relay gateway; The independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Smart Lock A integrates the reply messages and privacy processing results, encodes them to generate a video image data stream, and sends the video image data stream to the platform for decoding processing, thereby being able to ensure the protection of the face privacy of the property owner.

[0130] The above has detailed the structure, features and effects of the present invention according to the embodiments shown in the drawings. The above is only the preferred embodiment of the present invention, but the present invention is not limited to the scope shown in the drawings. Any changes made according to the concept of the present invention, or modified into equivalent embodiments with equivalent changes, still within the spirit covered by the specification and drawings, should be within the protection scope of the present invention.

Claims

1. A method for face privacy processing of an intelligent lock, characterized in that, The method comprises: Face detection and privacy processing: Smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, face privacy processing is performed; if the comparison fails, the face images that were not matched are sent to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other smart locks compare the face images that were not matched with the family member list of this lock. If the comparison is successful, face privacy processing is performed and the processing results are replied in unicast form; Message check and urging: Smart lock A counts multicast reply messages. If there are smart locks that have not responded, it sends a query request to the independent transit gateway; the independent transit gateway performs a health check and sends a urging reply message to the smart locks that have passed the health check; image integration and encoding: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes and generates a video image data stream containing descriptive metadata; smart lock A sends the integrated video image data stream to the platform for decoding processing.

2. The method according to claim 1, wherein The face detection and privacy processing include: Smart lock A detects multiple faces through video images, performs face segmentation and marks the face image ID and location coordinates; Calculate the similarity between the segmented face image and the family member list of this lock. If the similarity exceeds the threshold, replace the face image with the person ID. If the matching fails, the unmatched facial image and facial image ID will be sent to other smart locks in the community in multicast form.

3. The method according to claim 2, wherein The collaborative face privacy processing includes: Smart lock B receives the multicast message, extracts the unmatched face image and compares it with the family member list of this lock; If the match is successful, the facial image is replaced by the person ID, and the processing result is replied in unicast form; If the comparison fails, a unicast message will be sent to indicate that the match did not succeed.

4. The method according to claim 3, wherein The message checking and urging include: Smart lock A counts the multicast reply messages and checks whether there are any smart locks that have not responded. If there is a smart lock that has not responded, a query request is sent to the independent transit gateway; The independent transit gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check.

5. The method according to claim 4, wherein The image integration and encoding include: Smart lock A integrates the reply messages from other smart locks and the privacy processing results of this lock; Replace the face image with the person ID and leave the panoramic image blank; Generates a video image data stream containing descriptive metadata and embeds the metadata into the encoded data.

6. A system for intelligent lock face privacy processing, characterized in that, The system comprises: Detection module, used for face detection and privacy processing: Smart lock A detects multiple faces, performs face segmentation and recognition comparison; if the comparison is successful, performs face privacy processing; if the comparison fails, sends the unmatched face image to other smart locks in the form of multicast messages for collaborative privacy processing; Collaborative face privacy processing: After receiving the multicast message, other smart locks compare the unmatched face image with the family member list of this lock. If the comparison is successful, perform face privacy processing and reply the processing result in unicast form; Processing module for message checking and urging: Smart Lock A counts multicast reply messages. If there are smart locks that have not replied, it sends a query request to the independent relay gateway; the independent relay gateway performs a health check and sends a reminder reply message to the smart locks that pass the health check; Image integration and encoding: Smart Lock A integrates the reply messages from other smart locks and the privacy processing results of this lock, and encodes them to generate a video image data stream containing descriptive metadata; Smart Lock A sends the integrated video image data stream to the platform for decoding processing.

7. The system according to claim 6, characterized in that, The detection module is specifically configured to: Smart Lock A detects multiple faces through video images, performs face segmentation, and labels the face image ID and position coordinates; Calculate the similarity between the segmented face image and the list of family members of this lock. If the similarity exceeds the threshold, replace the face image with the person ID; If the comparison fails, send the unmatched face image and face image ID in multicast form to other smart locks in the community.

8. The system according to claim 7, wherein The detection module is specifically configured to: Smart Lock B receives the multicast message, extracts the unmatched face image, and compares it with the list of family members of this lock; If the comparison is successful, replace the face image with the person ID and reply with the processing result in unicast form; If the comparison fails, reply with the unmatched information in unicast form.

9. A storage medium, characterized in that, The computer program is stored in the storage medium, wherein the computer program is set to execute the method according to any one of claims 1-5 when running.

10. An electronic device, comprising a memory and a processor, characterized in that, The computer program is stored in the memory, and the processor is set to run the computer program to execute the method according to any one of claims 1-5.

Images